In part one and two of our series, we were able to rate the interaction of worthwhile features such as automatic waits, locators and reporting absolutely unproblematic. We also observed satisfactory teamwork between the two tools when it comes to the essential topic of debugging, as well as hooks and tags.
Where it gets dicey
But the cooperation is not so harmonious on all levels. Let’s now turn our attention to the sticking points of our setting and take a look at exactly where teamwork is put to the test.
Timeout: global
Playwright automatically sets a default timeout for actions such as .click(), .fill(), .clear() to 30 seconds. This can be changed individually in different places or globally in the playwright.config.ts. However, since our Cucumber Runner does not use the Playwright-Config, we instead set our own default timeout of 60 seconds globally for our Cucumber Runner, at the spot whereour hooks live:
e2e/src/support/common-steps.ts
setDefaultTimeout(60 * 1000);
Individually, we can also adapt this cucumber timeout, e.g. in the steps. For example, we can use the timeout attribute to turn down the cucumber timeout for this one step from 60 seconds to 5 seconds.
e2e/src/steps/sample.steps.ts
Then('the sample reveals the box of information', {timeout: 5000},async function () {...}
The problems begin when we define a cucumber timeout globally or in individual steps and the Playwright Runner is not used, as the default timeout of 30 seconds still applies globally to all actions of Playwright. We cannot change this globally or even simply turn it off because the location for such changes, the playwright.config.ts, is not being accessed. For us this means that, on one hand, the default timeout observed by Playwright cannot be changed, and on the other hand, all individually set timeouts from Cucumber only take effect if the set time is less than the default Playwright timeout. So if I set my cucumber timeout of the step to, for example, 40000 ms (i.e. 10 seconds more than the Playwright default timeout), the test still aborts after 30 seconds due to the default Playwright timeout.
Timeout: individual
Even though there is no global solution for the Playwright default timeout, there is a glimmer of hope: We have the option of setting individual timeouts for individual actions, which then also override the default timeout set by Playwright. For example, if I want to execute a .click() on an element which takes a longer time to load, I can set an individual timeout directly when calling this particular action:
await page.getByText("Suchen").click({timeout: 40000});
We observe a similar pattern in the timeout for the assertion library expect. For this purpose, Playwright has set a default timeout of only 5000 ms, globally changeable only in the playwright.config.ts. This can be shortened – but not extended – with a cucumber timeout. Luckily here, too, we have the option of shortening or extending the Playwright default timeout individually for a specific expect action:
await expect(page.getByTestId('testid'))
.toBeVisible({timeout: 10000});
Individually, this is a good solution for special cases, but we still lack easy handling of all timeouts in one place.
Soft assertions
Let’s stay with the expect assertions for a moment. Playwright also offers these as softAssertions. If a test breaks immediately if the expectation of an assertion is not fulfilled, softAssertions only indicate this for the time being, but allow the subsequent steps to continue running. For example, you can run several such assertions in succession. Playwright is used as follows:
await expect.soft(page.getByTestId('status')).toHaveText('Success');
The test does not stop at this point, but only reports an error message. This can reduce the number of tests in the case of several equal checks. Unfortunately, the softAssertions can only be used with the Playwright Runner, so we can’t use them in our setup. A simple replacement with cucumber is also not possible. If desired, another softAssertion library could of course be integrated, which then takes over this function.
Snapshots
Finally, let’s take a look at a visual detail. Playwright offers a snapshot match, which means that two snapshots are compared with each other side-by-side. This makes it possible to quickly check the durability of pages without changing any content. Playwright provides us with two helper methods. The first variant is as follows:
await expect(page).toHaveScreenshot();
In this variant a folder with the appropriate snapshot is automatically created during the first test run. When the test is called repeatedly, a new snapshot is compared with the one already saved. Alternatively, in the second variant:
await expect(value).toMatchSnapshot(snapshotName);
it is possible to compare a special screenshot, but also values or texts with a defined reference. However, both methods require integration with the Playwright Test Runner, which means that this comparison cannot be made with the help of the Cucumber Runner. A visual comparison on the part of Cucumber does not exist. The test automation engineer with his trained eye must therefore continue to take on this task, or alternatively look for an additional auxiliary tool.
Conclusion
In the three parts of our article series, we have examined the interaction between the BDD framework Cucumber and the automation tool Playwright in detail and at different levels. In conclusion, we can say that the two tools can generally be combined well with each other. The strengths of the respective tools are not curtailed. On the contrary, we benefit from Cucumber’s user-centered approach, the excellent comprehensibility of the requirements to be tested and the clear reporting, as well as from Playwright’s technical refinements, ease of use and transparent analysis options. In return, we are happy to accept that we sometimes have to opt for a variant or make additional configurations. Although there remains a hint of bitterness around timeouts, visual comparisons and soft assertions, the interaction of both tools is a suitable solution for carrying out acceptance tests efficiently and quickly.
