As is the case with any combination of technical tools, the question arises as to whether all parties involved can play to their strengths. A tool is only efficient if it is not hindered or restricted by the other tool. Therefore, today we take a closer look at the interaction and possible breaking points of two tools that are currently enjoying great popularity in acceptance tests. While working on a reliable software solution, the acceptance test tests the entire system in a different way than the other test stages: it examines logic and functionality from the user’s point of view; it must observe an adequate degree of human communication and requires an efficient technical implementation option. At this point, tools are combined that have to meet very different requirements and perspectives. First, let’s take a closer look at the two tools we use.
The powerful open source BDD tool Cucumber, used here in the Cucumber.js variant, primarily offers us an interface to map examples of system behavior in a language that everyone can understand. A common language is essential in the acceptance test, as it is the interface between business requirements and technical implementation. According to the four-eyes principle, the acceptance test uses its scenarios to check whether customer wishes and requirements are actually being met.
Scenario: The user will be informed if no results for the search term can be found
Given Dirk is logged in to the employee page
When Dirk searches for the surname 'Xo'
Then he gets a hint that no results were found
The Cucumber framework supports us with its parser mechanism, which translates each test step into an automated code step. In addition, Cucumber offers its own test runner, a tagging and filtering system as well as informative test reports.
If we then automate the previously agreed test scenarios, we want this to run as smoothly, quickly and efficiently as possible. That’s why we chose Microsoft’s relatively new automation tool Playwright, because Microsoft promises us exactly what we’re looking for: Playwright is fast, reliable and stable – without flaky tests, with automatic waits and retries, usable on any browser and platform, and with a sophisticated debugging and tracing system.
So, each tool seems to have its merits. But how well do they harmonize with each other and where does it become more difficult?
A wonderful collaboration
Upon first glance, the responsibilities seem relatively clear when using Cucumber.js and Playwright together. Nevertheless, there are certain areas where functionalities and solutions of both tools overlap. The most prominent overlap is the test runner itself. Both tools come with their o wn, but only one can be used. Since the Cucumber.js framework cannot be served by anyone other than the Cucumber Test Runner, we have to discard the Playwright Test Runner. Fortunately, this does not affect the following features of Playwright, which are detailed on the official Playwright website.
Browser: Mode & Fixtures
The automation tool supports different browsers and offers us both a headless and a headed mode:
e2e/config/launchOptions.ts
import { LaunchOptions } from "@playwright/test";
export const launchOptions: LaunchOptions = {
headless: false,
slowMo: 0,
};
In addition, with its so-called “fixtures” (browser, context, page), it offers the possibility to run individual tests in isolation from each other thanks to its own browser contexts. Once started, the browser can be used throughout the run.
Locators
Since you want to avoid flaky or false positive/negative tests when testing the GUI, the formulation of unique locators for the elements you are looking for is the most important craft. Playwright comes with an extensive library for this purpose, which provides us with various help methods and search mechanisms. These locators, which are advertised as best practice, are really the first choice for most cases and also behave very similarly to the visual perception of a human user. Most common searches include texts, labels or roles of elements. Through combinations and filter options, it is possible to select elements unambiguously.
await page.getByRole("button")
.getByText("Search")
.click();
Waits
Waiting for visibility and interaction of an element is another central point of any automation framework. Playwright is indeed a great way to accommodate us here. With the help of automatic retries and waits, the action capability itself is first checked. As example of this is checking whether the element is loaded, visible and activated. This saves us from explicit waits or assertToBeVisible. Thus, false positive/negative tests or breaking the test in the wrong place are also prevented at this point. Playwright automatically aborts the test with an error message if such a check is not passed.
Reporting
Regardless of the test runner, both tools have reporting options. Both are usable, you just have to choose one. In our case, we opted for reporting from the Cucumber environment. With plugins, the JSON report generated by Cucumber can even be converted into informative and descriptive HTML reports, which we can display in the browser – e.g. with the help of the cucumber-html-reporter, which has its own dashboard view, or the Java Jenkins plugin cucumber-reporter especially for Jenkins builds.
Outlook
After these promising basic functions, the next step is to take a closer look at the debugging options, as well as hooks and tagging options. And this much can be revealed: It will not remain as harmonious as before.
