Running Your First Test with Robot Framework and Docker

This is the first in a series of 3 articles. When you complete them all, you will have a functional and extensible, albeit basic, test automation solution for web applications. This solution will be prepared for inclusion in a CI/CD Pipeline, as well as for local execution.

The complete solution will use Robot Framework as a generic test automation tool, and use Selenium Webdriver to perform very basic browser automation. Two methods of parametrization of test runs will be introduced:

• parametrization via environment variables for parameters that are to be changed at runtime (for example: browser to use, or target system base URL)
• parametrization via YAML configuration files for parameters that are to be changed occasionally, but due to their nature, require aggregation in one place (for example: web elements selectors, or test data)

After you complete this article, you will have successfully created your first (empty) test case, and executed it using a test runner docker image.

Please note, that the article is meant to be quite hands-on, and even though I will do my best to explain the concepts being introduced, I cannot effectively describe all possible aspects of the solution in detail, as I would be doubling the excellent documentation of the projects that are in use. If you'd rather just look at the code, feel free to clone this companion repository. Code relevant to this specific article is available under a specific tag.

Who is this for

I assume that you:

• Want to start automating your website or web app testing
• Haven't yet decided on the technology, but you'd value tried and tested over new, shiny and possibly broken
• Value good documentation
• Need to deliver good results, quickly

There are no specific assumptions about your:

• Technical affinity
• Testing experience

Motivation

During my career, I've seen multiple teams struggle with test automation solutions on all levels, including web testing automation. A quick Google Search returns a Quora thread with list of usual challenges related to this area of our daily work. In short, as a test automator, you will most definitely face those questions:

• How do I know what should be automated?
• How do I ensure that the automation solution correctly identifies relevant page elements?
• How do I run my tests against multiple browsers, or more broadly - in multiple environments?
• How do I make sure that the tests convey real users' interactions with my system?
• How do I make the tests run as rapidly as possible? How can I run my tests in parallel?

If you were to use Robot Framework as your automation solution, I'd say you were on a good path to solving almost all of those problems. How? Every one of those could be possibly its own article! But first...

Why Robot Framework

Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has easy-to-use tabular test data syntax and it utilizes the keyword-driven testing approach.

(Robot Framework Official Webpage)

I admit that Robot Framework is my personal preference when it comes to test automation, but I hope to at least intrigue you enough to give it a go. Here are some of the reasons why I like and recommend using Robot Framework:

• It is a generic test automation framework that does not make any assumptions about your application, and can be used to test anything - APIs, webpages, mobile applications, desktop applications or even device drivers.
• Test specifications are simple text files in natural language. They make it easy to reason about the test cases, and discuss their content and coverage with interested project stakeholders. This also means that you can write test cases in your preferred language.
• It's easily extendable, so it can act as a glue between multiple testing tools
• It provides a nice report and log.
• It is open source and free.
• There's a growing community available.
• It has excellent documentation, both for how to use it, and on the specific test keywords.
• It has a concept of critical and non-critical tests. As some test levels are more brittle than others, and delivery of fixes may take more time, it's good to know that we can report on a failing assertion, yet still accept the test results as valid.
• It can talk to your system via any interface that you can imagine, via browsers, APIs, SSH, direct database connections, and if all else fails - by virtually looking at your screen.
• It lets you organize your tests in any way you find appropriate by moving files around.
• It lets you easily separate your tests from your test data, and even generate test data on the fly, or have it sourced from external systems.
• It lets you execute your tests in any order or way you find necessary, and it can generate a unified test run report afterwards.

Why Selenium

Even though there have been new web test automation frameworks popping up (see: TestCafe or Cypress), Selenium remains one of the best choices to consider, if you have a website or webapp to automatically verify. What is Selenium? In words of its authors:

Selenium is a portable framework for testing web applications Selenium provides a playback (formerly also recording) tool for authoring functional tests without the need to learn a test scripting language (Selenium IDE). It also provides a test domain-specific language (Selenese) to write tests in a number of popular programming languages, including C#, Groovy, Java, Perl, PHP, Python, Ruby and Scala. The tests can then run against most modern web browsers. Selenium deploys on Windows, Linux, and macOS platforms. It is open-source software, released under the Apache 2.0 license: web developers can download and use it without charge.

(Selenium Wikipedia Page - emphasis mine)

The important aspect for us, is that Selenium allows us to use most mainstream browsers in uniform way in order to automate web testing. Robot Framework comes equipped with SeleniumLibrary, that can be used to interact with a remote SeleniumHub. For our purposes, a hub can be brought up with help of Docker (again), but in the future you can either roll your own, or use one of many services like BrowserStack or SauceLabs.

Prerequisites

In order to complete this article's goals, you will need the following:

• Docker installed and running. For development, I suggest that you can run docker commands without having to ask for superuser privileges every time - see here for details, but note that this is considered insecure..
• bash installed and running -- use either a Linux distro, a VM or - if you're running Windows - WSL.
• code editor capable of editing Robot Framework files, as well as script files. I recommend using VS Code.

I assume that you are somewhat familiar with code editing and working on the command line.

Writing and understanding a basic test file

OK, so in order to run a first test case, we need a first test case. Robot Framework allows us to write test cases in simple text files using natural language. You may be familiar with this concept if you have ever worked with Cucumber. This is important, because from the perspective of test automation specialist, it helps focus on business oriented aspects of the tests, instead of technical aspects. In other words, it's more important to say that a user account must exist, than to describe, step by step, how to make one.

Let's say that we want to test this testers-friendly page in both Chrome and Firefox. In order to do that, we will be taking baby steps. First, we need to get Robot Framework going, and running an empty test suite. This will be a file called initial_test_suite.robot in directory tests/ of our project:

tests/initial_test_suite.robot

Robot Framework uses the filesystem as a natural way of organizing test suites into hierarchical structures. Each directory that contains test text files may be recognized as a test suite itself, with its own respective setup, teardown, or metadata. Robot Framework will treat tests/ as a test suite (collection of tests), but initial_test_suite.robot will also be a test suite. This is an important concept - the test suites can and will be nested.

The usual Robot Framework file extension is .robot. A .robot file can be treated by Robot Framework as either a test suite or a resource file. They are more or less the same, with the important difference being their purpose: test suites contain test cases and are meant for execution, whereas resource files contain only keywords for shared use in multiple test suites. Resource files can be referenced by other resource files, or by test suites, but test suites cannot be referenced by any other file.

The structure of the .robot file is understandable, but also very flexible. It contains one or more sections, those being:

• Settings
• Variables
• Test Cases - only in test suites
• Keywords

Each section contains keywords, which you can treat like functions in other programming languages. You can either define keywords in the same file, or import either resource files or external libraries that provide you with additional keywords. A keyword may accept additional arguments, but we’ll investigate it deeper in next series installment.

Text written without indentation under Test Cases and Keyword headers is treated as test case name and keyword name respectively, and test case content and keyword content must be indented by one level - at least two spaces. Using this knowledge, we can write the simplest test suite (tests/initial_test_suite.robot) to get our setup going:

*** Test Cases ***
This test case does nothing
   #This is a comment.
   #Remember to indent the test case content with at least two spaces
   No operation

No operation is a keyword from BuiltIn Library, and - as the name suggests - we don't have to import anything to use those keywords.

What's going on here

This test case does nothing is the name of the test that we want to execute. As you see, it contains exactly one keyword to execute. This keyword, No Operation, will do nothing, just pass the test execution.

Running the test and creating a test runner

In order to actually run the test suite, we can go one of two ways: install Robot Framework locally, or use docker image to create a container with our test execution. Let's go the Docker way, simply because it is more robust (it won't get affected by the works on my machine syndrome). Fortunately, most of the legwork has already been done for us in the robotframework/rfdocker project, and we can (for now) call the latest rfdocker image:

docker run \
 --rm \
 --volume "$PWD/tests":/home/robot/tests \
 --volume "$PWD/results":/home/robot/results \
 robotframework/rfdocker:latest \
 tests

This is quite a lot to unload, and I'll refer you to docker documentation for all possible options, but here's a rundown:

• docker run: use docker to execute a specified image as a container
• --rm: don't store the container after execution had ended
• --volume "$PWD/tests":/home/robot/tests: link local directory tests/ to the container directory /home/robot/tests. /home/robot is the default location where the Robot Framework will be invoked when container is run.
• --volume "$PWD/results":/home/robot/results: similar, link local directory results/ to the container directory /home/robot/results. This is the default test log output directory. By linking it to our machine, we ensure that the execution logs will be saved to our development machine, and not lost with container deletion.
• robotframework/rfdocker:latest: specifies the docker image for execution. latest is fine for development purposes, but you will want to link a specific image version for your production environment
• tests: specifies the name of the test suite to run. So the default Robot Framework invocation directory /home/robot concatenated with tests gives us /home/robot/tests - which is exactly the directory we had linked with our local test suite content.

I hate to repeat myself, so let's just pack it into a run.sh script to avoid early onset of RSI:

#!/usr/bin/env bash

docker run \
 --rm \
 --volume "$PWD/tests":/home/robot/tests \
 --volume "$PWD/results":/home/robot/results \
 robotframework/rfdocker:latest \
 tests

Now for last adjustments, let's make it executable, and in case you were using git for your version control (which you should), we'll add the results/ directory into our .gitignore file:

chmod +x run.sh
echo results/ >> .gitignore

We can finally call run.sh and enjoy the magnificent results of our first, unproductive test case:

> ./run.sh
============================================
Tests
============================================
Tests.Initial Test Suite
============================================
This test case does nothing         | PASS |
--------------------------------------------
Tests.Initial Test Suite            | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
============================================
Tests                               | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
============================================
Output:  /home/robot/results/output.xml
Log:     /home/robot/results/log.html
Report:  /home/robot/results/report.html

You will notice that the results/ directory has appeared in your working directory, and there are 3 files in it:

• output file -output.xml - contains machine-readable test results summary. Review an example output file.
• report file - report.html - this is a high-level overview of your test execution. For ease of finding out if test run had been successful, it has either bright green or bright red background. See an example report file.
• log file - log.html - this is the interesting one, as it lets you dive deep into all the details of test suite execution. See an example log file.

Congratulations, you have successfully executed your first Robot Framework Test Case!

Summary

• We've implemented a basic Hello World-type test case in Robot Framework
• We've successfully executed it using Robot Framework in a Docker container
• We've codified the test runner as a script, so it could be at this stage placed in CI/CD Pipeline and act as entry point for future tests.

In the next installment, we'll enhance our test and add some parameters to it, via files stored in the project, and via environment variables.