Parameterizing Your Robot Framework Tests
This is part 2 of 3 of the Getting Started with Robot Framework Series. You should read the first part, but also the companion article on why parameterizing test cases is important before continuing, as it contains the basis for work done in this article.
If you're too impatient to read the whole thing (it is quite lengthy!) or just want the juice, here's the code.
Goal of this installment
After you have worked through this part of the series, you will have
- An improved test runner
- A way to include new Resource Files in your tests
- Some custom keywords that you can use in multiple test cases
- A place to store grouped variables used in your tests
- A test case that demonstrates various ways to parameterize the test cases
There is one goal that will not be achieved at this moment, and that is an actual Selenium-backed automation. This comes in the next part of the series.
A few definitions
- Test Driver is a software component or test tool that can control the component or system under test.
- Test Environment is the hardware, instrumentation, simulators, software tools, and other support elements needed to conduct a test.
In our setup, Robot Framework acts as the test driver which is placed in the test environment.
As described in the companion article we distinguish various types of data that our test may be using:
Example Variable | Variable type | Type description
- | - | -
browser, host | operational data | describe the context in which the tests will be executed: which environment, which proxy, what browser, what locale, etc.
login-page-url, main-site-url, username-field, password-field, login-button | structural data | describe the structure of application, usually used to describe User Interfaces
valid-username, valid-password | actual test data | describe the data that actually differs on case-to-case basis. This is what you think differentiates specific test cases and groups SUT behaviors
Test Case Definition
We had finished our last work with a plan to test this testers-friendly website and a rather unimpressive test case:
*** Test Cases ***
This test case does nothing
#This is a comment.
#Remember to indent the test case content with at least two spaces
No operation
There are a couple of things that we should do
- Convert it to a test case that at least describes an actual workflow (what to test)
- Create scaffolds of keywords that describe how to achieve the result (how to test)
- Extract parameters from this test case so that we can easily manage and extend it
In the end we will have implemented a (not yet fully functional) test case that does the following:
- Opens web page
http://automationpractice.com/index.php - Searches for fixed term
dress - Checks the name of the first search result item
- Opens the first result from the search results
- Checks that a result pages opens
- Checks that the breadcrumb contains the same name as the first search result
Let's write this up as robot framework test case:
*** Test Cases ***
Search and verify result breadcrumb
Open web page http://automationpractice.com/index.php
Search for "dress"
Remember title of 1st search result item
Open 1st search result
Verify that a product page is now open
Verify that the breadcrumb contains the same name as the 1st search result title
This shows us a big advantage of Robot Framework - the test case is an actual plain English description of the operations to be performed. At this stage it fails to execute correctly, because we have instructed Robot Framework what is to be done, but not how to do it.
To run the test, type the following into your bash console:
./run.sh
Example results:
==============================================================================
Tests
==============================================================================
Tests.Initial Test Suite
==============================================================================
Search and verify result breadcrumb | FAIL |
No keyword with name 'Open web page http://automationpractice.com/index.php' found.
------------------------------------------------------------------------------
Tests.Initial Test Suite | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Tests | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: /home/robot/results/output.xml
Log: /home/robot/results/log.html
Report: /home/robot/results/report.html
Initial keywords definitions
Let's fix it by providing (still empty) keyword definitions. A keyword is the plain English instruction that we want to describe in detail - we've used them to tell Robot Framework about our test case.
To define a new keyword we introduce a new section in our test file: *** Keywords ***. In this section, every line starting without indentation is a definition of a new keyword. Every indented line afterwards describes part of the keyword. For now, we can simply use the built-in placeholder: No operation.
*** Test Cases ***
Search and verify result breadcrumb
Open web page http://automationpractice.com/index.php
Search for "dress"
Remember title of 1st search result item
Open 1st search result
Verify that a product page is now open
Verify that the breadcrumb contains the same name as the 1st search result title
*** Keywords ***
Open web page http://automationpractice.com/index.php
[Tags] not yet ready
No operation
Search for "dress"
[Tags] not yet ready
No operation
Remember title of 1st search result item
[Tags] not yet ready
[Documentation] Extract a value from the webpage and make it available in the remainder of the test
No operation
Open 1st search result
[Tags] not yet ready
No operation
Verify that a product page is now open
[Tags] not yet ready
No operation
Verify that the breadcrumb contains the same name as the 1st search result title
[Tags] not yet ready
No operation
Of course, after this change, the "test" passes:
==============================================================================
Tests
==============================================================================
Tests.Initial Test Suite
==============================================================================
Search and verify result breadcrumb | PASS |
------------------------------------------------------------------------------
Tests.Initial Test Suite | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Tests | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output: /home/robot/results/output.xml
Log: /home/robot/results/log.html
Report: /home/robot/results/report.html
(Remember: in this part of the series we're only parameterizing the test and making sure that we can execute it in various ways)
Parameterizing simple keywords
A very nice feature of Robot Framework allows you to embed placeholders in the names of the keywords, which leads to dynamic discovery of applicable keywords. For example, consider the URL address that needs to be opened at the beginning of the test. It may seem fixed - at least in our example. Under usual circumstances you wouldn't be testing a production system. Instead you would have one or more test systems, maybe even a local deployment, so this keyword simply cannot have the URL embedded in it. Let's write it with a placeholder instead:
Open web page ${url}
[Tags] not yet ready
Log ${url}
After running our test suite now, and investigating the results/log.html file, you will find that the value has indeed been placed correctly:
This will allow us to call it with any URL. And as a matter of fact, we should also immediately tell Robot Framework, that the URL is not fixed:
*** Test Cases ***
Search and verify result breadcrumb
Open web page ${SUT.baseurl}/index.php
Search for "dress"
Remember title of 1st search result item
Open 1st search result
Verify that a product page is now open
Verify that the breadcrumb contains the same name as the 1st search result title
We could go even further, and use a placeholder to semantically describe the page we want to open:
*** Variables ***
${main page} ${SUT.baseurl}/index.php
*** Test Cases ***
Search and verify result breadcrumb
Open web page ${main page}
Search for "dress"
Remember title of 1st search result item
Open 1st search result
Verify that a product page is now open
Verify that the breadcrumb contains the same name as the 1st search result title
How will we inform the Robot Framework what the ${SUT.baseurl} is? I'd wager that using an environment-specific config file is the way to go. We'll create two: env/dev.yaml and env/prod.yaml, for our development and production environments correspondingly.
env/dev.yaml
SUT:
baseurl: http://localhost:8080
env/prod.yaml
SUT:
baseurl: http://automationpractice.com
Adjusting the runner file to include the environment description
Now that we have some environment-specific configuration files, we will slightly adjust the runner file. First of all, we'll tell Docker to just use the whole working directory as the /home/robot directory, so that all future variable files will be automatically included. Second of all, we'll tell Robot Framework to use a specific environment description file:
#!/usr/bin/env bash
docker run \
--rm \
--volume "$PWD":/home/robot/ \
robotframework/rfdocker:latest \
--variablefile "env/$ENV.yaml" \
tests
To run it now, we need to pass the information about which environment the tests are running against. You can do it by either setting environment variable in your system (CI service usually does this to some capacity), or, when testing things manually, you can just prefix your command with the environment variable:
ENV=dev ./run.sh
This will now load the content of ./env/dev.yaml file, execute the tests and... fail:
[ ERROR ] Processing variable file '/home/robot/env/dev.yaml' failed: Using YAML variable files requires PyYAML module to be installed. Typically you can install it by running `pip install pyyaml`.
==============================================================================
Tests
==============================================================================
[ ERROR ] Error in file '/home/robot/tests/initial_test_suite.robot': Setting variable '${main page}' failed: Resolving variable '${SUT.baseurl}' failed: Variable '${SUT}' not found.
Tests.Initial Test Suite
==============================================================================
Search and verify result breadcrumb | FAIL |
Variable '${main page}' not found.
------------------------------------------------------------------------------
Tests.Initial Test Suite | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Tests | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================
Output: /home/robot/results/output.xml
Log: /home/robot/results/log.html
Report: /home/robot/results/report.html
We now lack support of the YAML format in our docker runner.
Add support for YAML to Robot Framework
According to documentation, we need to somehow add custom module to our docker image. This is a good thing, we'll need it anyway in the future, to add Selenium support. So let's get down to it. We need to:
- build off of the official
robotframeworkimage - install some additional requirements (this is usually done using a
requirements.txtfile andpipcommand) - build our docker image
- use the newly created docker image instead of official one
We will create a Dockerfile to describe our changes, and requirements.txt to describe our - duh - requirements:
requirements.txt
To learn more about this format, refer to pip documentation:
pyyaml
Dockerfile
FROM robotframework/rfdocker:latest
COPY requirements.txt /requirements.txt
RUN pip install -r /requirements.txt
run.sh
Here we're telling docker to first take instructions from Dockerfile and convert them into a functioning image under name customrf - and then we're using this custom image to execute our tests. For more information on building Docker images, see here.
#!/usr/bin/env bash
docker build -t customrf .
docker run \
--rm \
--volume "$PWD":/home/robot/ \
customrf:latest \
--variablefile "env/$ENV.yaml" \
tests
Execute tests
Nothing changes in our execution, but the results are completely different:
ENV=dev
ENV=dev ./run.sh
Sending build context to Docker daemon 612.4kB
Step 1/3 : FROM robotframework/rfdocker:latest
---> 78d04f0de2cd
Step 2/3 : COPY requirements.txt /requirements.txt
---> Using cache
---> 87a607257be2
Step 3/3 : RUN pip install -r /requirements.txt
---> Using cache
---> 40ef6e53a35e
Successfully built 40ef6e53a35e
Successfully tagged customrf:latest
==============================================================================
Tests
==============================================================================
Tests.Initial Test Suite
==============================================================================
Search and verify result breadcrumb | PASS |
------------------------------------------------------------------------------
Tests.Initial Test Suite | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Tests | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output: /home/robot/results/output.xml
Log: /home/robot/results/log.html
Report: /home/robot/results/report.html
ENV=prod
ENV=prod ./run.sh
Sending build context to Docker daemon 612.4kB
Step 1/3 : FROM robotframework/rfdocker:latest
---> 78d04f0de2cd
Step 2/3 : COPY requirements.txt /requirements.txt
---> Using cache
---> 87a607257be2
Step 3/3 : RUN pip install -r /requirements.txt
---> Using cache
---> 40ef6e53a35e
Successfully built 40ef6e53a35e
Successfully tagged customrf:latest
==============================================================================
Tests
==============================================================================
Tests.Initial Test Suite
==============================================================================
Search and verify result breadcrumb | PASS |
------------------------------------------------------------------------------
Tests.Initial Test Suite | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Tests | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output: /home/robot/results/output.xml
Log: /home/robot/results/log.html
Report: /home/robot/results/report.html
Adding more semantic variables to keywords
We now can repeat the process of parameterizing keywords to enhance our other keyword: the one that enters a search criterion.
Search for "dress"
[Tags] not yet ready
No operation
What is "dress" in this case? It's a search term, so we'll call it that:
Search for "${search term}"
[Tags] not yet ready
Log ${search term}
The quote characters in this case are not required, but they are a useful trick. They provide a good visual (and programmatic) discriminator for the boundaries of the variable. The way Robot Framework tries to identify what's a variable in the name, and what isn't, it runs into risk of overshooting with the matching mechanism. Imagine, that you didn't use the quotes, and later added a keyword Search for ${search term} using the search bar at the bottom of the page. Robot Framework will get confused about which keyword to use, hence the quotes that help him decide.
As for now, that's all that we'll do with this keyword.
Parameterizing keywords with complex placeholders
We can change it to Remember title of search result item number 1. It will increase readability of the test case, but there’s also more that we can achieve: we can use initial input data validation to tell Robot Framework "Hey, only numbers here, please":
...
*** Test Cases ***
Search and verify result breadcrumb
Open web page ${main page}
Search for "dress"
Remember title of search result item number 1
Open 1. search result
Verify that a product page is now open
Verify that the breadcrumb contains the same product name as the search result no. 1 title
*** Keywords ***
...
Remember title of search result item number ${number:\d+}
[Tags] not yet ready
[Documentation] Extract a value from the webpage and make it available in the remainder of the test
Log ${number}
Open search result number ${number:\d+}
[Tags] not yet ready
Log ${number}
Verify that the breadcrumb contains the same product name as the title of search result number ${number:\d+}
[Tags] not yet ready
Log ${number}
...
This should still run, and we should nicely retrieve ${number} from the test case:
Structuring of resources
Robot Framework has a method of structuring the keywords for easy reuse. Namely, you can provide a resource file which contains keyword definitions, and include it in multiple test suites. It is a good idea to follow modularization principles and provide rather specifically tailored resources.
We will now split the operations available in certain pages to page-oriented resources. First, let's create a proper directory structure (don't mind the components subdirectory, we'll get there with time):
#> tree resources
resources
├── components
└── pages
├── main_page.robot
├── product_page.robot
└── search_results.robot
2 directories, 3 files
Resource pages are simply robot framework files that lack the Test Cases section. Now we will split our existing keywords into specific resources:
resources/pages/main_page.robot
*** Keywords ***
Search for "${search term}"
[Tags] not yet ready
Log ${search term}
resources/pages/search_results.robot
*** Keywords ***
Remember title of search result item number ${number:\d+}
[Tags] not yet ready
[Documentation] Extract a value from the webpage and make it available in the remainder of the test
Log ${number}
Open search result number ${number:\d+}
[Tags] not yet ready
Log ${number}
resources/pages/product_page.robot
*** Keywords ***
Verify that the breadcrumb contains the same product name as the title of search result number ${number:\d+}
[Tags] not yet ready
Log ${number}
Verify that a product page is now open
[Tags] not yet ready
No operation
Putting it together
In order to tell Robot Framework that he has to include additional directories in Library search path, we need to include additional line --pythonpath "resources/" in our runner script:
#!/usr/bin/env bash
docker build -t customrf .
docker run \
--rm \
--volume "$PWD":/home/robot/ \
customrf:latest \
--variablefile "env/$ENV.yaml" \
--pythonpath "resources/" \
tests
And we have to tell the test suite where to look for all the keywords that we have just removed from our test suite. We'll do it by providing a Settings section in the test suite file:
*** Settings ***
Resource pages/main_page.robot
Resource pages/product_page.robot
Resource pages/search_results.robot
...
This is enough for our test suite to be passable, but there's one trick that I had learned from one of the Robot Framework experts, which definitely improves the readability of the test suite. The trick is to include information about the source of the keyword in the line where it is used. If we include this trick, we'll get the following test suite file:
*** Settings ***
Resource pages/main_page.robot
Resource pages/product_page.robot
Resource pages/search_results.robot
*** Variables ***
${main page} ${SUT.baseurl}/index.php
*** Test Cases ***
Search and verify result breadcrumb
Open web page ${main page}
main_page.Search for "dress"
search_results.Remember title of search result item number 1
search_results.Open search result number 1
product_page.Verify that a product page is now open
product_page.Verify that the breadcrumb contains the same product name as the title of search result number 1
*** Keywords ***
Open web page ${url}
[Tags] not yet ready
Log ${url}
After running the test suite now, we'll get an additional perk in the log.html - all the resources will be included in the log:
Setting test metadata from environment variable
Since we're already at defining Settings of the test suite, how about learning how to access environment variables directly in Robot Framework? We can do this to actually provide a bit of useful information for our test, namely shorten the time needed to find out in which environment the test had been executed. Let's add the following line to our settings:
*** Settings ***
Metadata Environment %{ENV}
...
Note the % instead of $ before in the variable name. It means that the variable comes not from Robot Framework, but from the environment. We do however run into a pickle here: since our runner is a Docker container, it does not automatically inherit the shell's environment. We have to explicitly tell Docker to set specific variables, and then Robot Framework will be able to access them.
This was not the case when we were telling it which environment configuration file to use - that had been resolved at the run.sh runtime, so Docker got told to use specifically --variablefile "env/prod.yaml".
We can tell Docker the value of environment variable to be passed into the container, by adding the following line in the run.sh:
-e ENV="$ENV"
The run.sh now looks like this:
#!/usr/bin/env bash
docker build -t customrf .
docker run \
--rm \
--volume "$PWD":/home/robot/ \
-e ENV="$ENV" \
customrf:latest \
--variablefile "env/$ENV.yaml" \
--pythonpath "resources/" \
tests
This produces the following bit of metadata in the log file:
Summary
At this moment we have the following:
- Custom Docker-based test runner (
Dockerfile) - Way to install additional modules to the test runner (
requirements.txt) - Specific test case
- Environment-specific operational parameterization via a combination of environment variables and yaml-files (
env/*.yaml) - Custom keywords and parameterized keywords
- Resources for keywords reuse
- Reporting of metadata and access of environment variables
What's coming next
In the next part of the series, we'll:
- Finally get some actual verification done!
- Bind Selenium to our test script
- Add Setup and Teardown sequences to the test data
- Finalize the page objects by using additional yaml files
