Let’s start by going back to the promising findings from the first part of our analysis of the interactions between Cucumber and Playwright: Browser-specific features, the targeted finding of elements with locators, as well as automatic waiting for the interoperability of elements are three strengths of Playwright, which we can also fully use with Cucumber. At the same time, Cucumber’s reporting system can still be used on demand without any problems.
Slight differences
Now let’s take a closer look at Playwright’s features and benefits that are directly or indirectly influenced by the test runner. As already mentioned in the first part of our analysis, we do not use the Playwright Test Runner in combination with Cucumber, which also eliminates the need to read the in-house playwright.config.ts file. This is usually where all the configurations required for the runtime are set: retry repetitions, timeouts for actions, navigation and assertions, selection of scenarios to run, skip and fail of individual tests, browser mode and much more. Of course, there is also a config file for Cucumber, the cucumber.js, in which we can also make runtime configurations. Nevertheless, both frameworks come dangerously close here.
Debugging: Inspector
The first critical edge case is debugging. Playwright comes with a whole range of debugging and monitoring capabilities that are a huge help in preventing test code bugs or finding “real” bugs, as well as creating tests. This is a very comprehensive inspector which includes step-by-step debugging, examining elements, as well as trying out and even generating locators. The Inspector is started with the simple prefix PWDEBUG=1 in the command line. The test then runs automatically in headed mode, in a separate, dedicated inspector window. Alternatively, an await page.pause(); stops at a desired point in the code, then opens the inspector, allowing for debugging. If the default setting is not set to headed in this case, the --headed flag must be added to the test start on the command line.
With the Inspector you can now go step by step through the test, simultaneously comparing the code with the changes in the GUI. This also works flawlessly with Cucumber. But be careful with the timing: If the inspector is started with the variable PWDEBUG, Playwright automatically sets the timeout to “0” (which is not the case with page.pause(), for example) in order to allow an unlimited inspection. However, since we have also set a default timeout in Cucumber, this now still applies, as it cannot be overlaid by Playwright. The solution is either to increase the ‘cucumber timeout’ or to manually switch it off for debugging.
Another function of the Inspector is Record: In this mode, Playwright records manually executed actions and generates code for them at the same time. This can then simply be transferred to your own repo via copy-and-paste. This function can be used without issue in the Cucumber setup and does not require any further settings.
Debugging: Trace
In addition to the Inspector, there is another GUI tool from Playwright: Trace. This records all actions, including snapshots, into a file, which can later be viewed again with the Trace Viewer. The tool shows both the code of the respective location, the snapshot and the log of the actions themselves, as well as the log of the console and the network. With the command show-trace the tool opens, specifying the respective file name, directly at the searched location.
npx playwright show-trace '.\src\reports\traces\The user is informed if no results are found for the search term.zip'
The pre-populated Playwright config handles the trace as the default setting. However, since the config is not read, an additional configuration is required in our setup. In the Before step, the trace must be started as follows:
e2e/src/support/common-steps.ts
Before(async function (this: ScenarioWorld, scenario){…
await this.context.tracing.start({ screenshots: true, snapshots: true });
…}
and logically closed again in the After step:
e2e/src/support/ common-steps.ts
After(async function (this: ScenarioWorld, scenario){…
await this.context.tracing.stop({ path: `path/${scenario.pickle.name}.zip`});
…}
With this small intervention, we can now use the trace tool without issue.
Hooks
Let’s take a quick look at hooks and tagging. Hooks, such as BeforeAll(), After() etc., which are important for our test setup, can be set at the test runner level on both Cucumber and Playwright. Since we use the Cucumber Test Runner, we can simply take the corresponding hooks from Cucumber.js.
Tags
As far as tagging is concerned, it appears as if Playwright basically offers its own system. On the one hand, the name of the file or the test itself can be used to trigger individual or multiple tests:
npx playwright test $filename
On the other hand, Playwright also offers a kind of keyword search for this:
npx playwright test --grep “search term”
These options are only available to us if we also use the Playwright Test Runner. However, in our case, we are using Cucumber’s tagging system. To do this, we specify in the cucumber.js which tags are to be controlled and then provide our tests in a Gherkin file with a corresponding annotation, such as “@example”.
Outlook
Even though Playwright and Cucumber seem to be vying for supremacy in some places, both tools still get along well and allow an efficient solution with little effort. However, in the last part of our analysis, we will really get down to the nitty-gritty and examine various possible weaknesses in this cooperation.
