we blog about software_

Introduction

With automated end-to-end tests, you can not only find bugs, but also regularly check if your software is compliant with security standards. Automation brings several advantages:

• Automated security tests provide reliable verification that security features are working as intended.
• They help keep security mechanisms stable during further development and detect unwanted regressions at an early stage.
• Writing automated tests allows you to look at your software from the perspective of potential attackers.

In this article, we use concrete examples to show how Playwright can be used to reliably test security-relevant aspects such as Content Security Policy (CSP), clickjacking or Cross-Site Request Forgery (CSRF).

Approach: Playwright end-to-end security testing

We will now focus on reviewing selected security aspects using automated end-to-end testing. These tests can be implemented alongside end-to-end feature tests as they can run in the same pipeline as these “normal” tests. Therefore, their development feels like the development of application feature tests. To show you how to check some aspects with the help of Playwright, we shall use an example which is relevant to CSP (Content Security Policy). The CSP is sent in the header of an HTML response, and it is configured during development of the frontend. If you are therefore intending to check the CSP, it is a good idea to call up the page as part of a test and perform the checks there. Playwright is currently the most common tool for end-to-end testing of a web application. By and large, the same approaches and methods can be used for security testing as are used for end-to-end testing for new features. In our tests for the CSP, we want to check various aspects.

Content Security Policy Review

The first aspect concerns simple access to the page being checked. The first thing we want to do is make sure that no CSP is being violated by the existing implementation. Therefore, we enter the page and check that no warning appears in the browser’s console. With a small helper function, we can capture browser console error messages produced during our Playwright test and store them in an array. We simply pass the page and the target array to the function, and its implementation appends any console errors to that array as they occur.

function logBrowserErrors(page: Page, errors: string[]) {
  page.on("console", (messsage) => {
    if (messsage.type() === "error") {
      errors.push(messsage.text());
    }
  });
}

Therefore, after calling up our page to be checked, we can validate that no CSP warnings or other error messages were triggered on the page. The check can be done using Playwright’s expect function.

expect(errors).toHaveLength(0);

When Playwright calls up the page, we also get a response to this call. This contains the CSP attributes in the header. We write these values to a so-called validation file, which is filled with the current CSP attributes when the test is run for the first time. These values must initially be critically checked for the expected values. If there are deviations from the expected values, the CSP must be adjusted so that the values in the validation file match the expected values.

Once the validation file has been released each subsequent run of the test, be it run locally or in a pipeline, compares the contents of the file to the obtained attributes. If a deviation is detected, the test fails. In this way, all changes to the CSP are reliably detected. If you plan to make changes to the CSP, the file can be adapted. In all remaining cases it is checked why the CSP has been changed, and it can be decided whether the change needs to be reversed or whether it can be kept.

Here’s an example of what the contents of such a validation file look like:

{
  "cspHeaderValues": [
    "default-src 'self'",
    "connect-src 'self'",
    "script-src 'nonce-[NONCE]' 'strict-dynamic' 'wasm-unsafe-eval'",
    "style-src-elem 'self' 'nonce-[NONCE]'",
    "style-src-attr 'unsafe-inline'",
    "img-src 'self' blob: data:",
    "font-src 'self' data:",
    "object-src 'none'",
    "base-uri 'self'",
    "form-action 'self'",
    "frame-ancestors 'none'"
  ]
}

We have masked the nonce values in this file because they are regenerated in each run and therefore the test cannot test for a concrete nonce value.

async function validateCSPData(
  response: Response,
  page: Page,
) {
  const cspHeaderValues =
    (await response.allHeaders())["content-security-policy"] ?? "";
  if (cspHeaderValues === "") {
    throw new Error("CSP must not be empty.");
  }
  const hasMetaCSP = await checkMetaCSP(page);
  expect(hasMetaCSP).toBeFalsy();
  const snapshot: Record<string, string[]> = {};
  snapshot.cspHeaderValues = cspHeaderValues
    .split(/;\s*/)
    .filter((str) => str !== "");
  await compareActualWithValidationFile(snapshot);
}

In the validateCSPData method shown, you can see our implementation for validating the CSP attributes. All we have to do is pass the page and the response of the page’s call to the method. The method extracts the proportion which affects the CSP from the response. In an initial validation, we make sure the CSP is not empty. We then run another check and validate that there are no meta CSP attributes in the HTML part of the response, as we have decided not to allow meta CSP attributes and we must check to avoid conflicts between the CSP in the header and in the meta-attributes. At the end of the method, we format the CSP attributes and pass them to our method, which compares the values with those in the file mentioned above.

Check CSP Warning

In a further step, we manipulate the HTML part of our page to be checked in order to verify that the expected CSP warnings appear in the browser’s console. In our example we add the following line to the HTML body of the page:

<script src="https://bad.test/evil.js" async=""></script>

This manipulation simulates an attack via XSS (Cross-Site-Scripting). In such an attack, “malicious code”, usually in the form of JavaScript, is injected into a website. If the code were to be executed, sensitive data could be tapped. Therefore, it is important to check that if code were to be injected into the page, it would not, under any circumstances, be executed. We manipulate the HTML body using the route method, which we apply to Playwright’s page object:

async function setupRouteWithModifiedBody(
  page: Page
) {
  await page.route(
    page.url(),
    async (route) => {
      const response = await route.fetch();
      let bodyForModification = await response.text();
      bodyForModification = bodyForModification.replace(
        "</body>",
        `<script src="https://bad.test/evil.js" async=""></script></body>`,
      );
      await route.fulfill({
        response,
        body: bodyForModification,
      });
    }
  );
}

In this method, we manipulate the call to the page to be checked. We’ll apply the route method to the URL of the page, manipulating the HTML body in the process. In the route method, we specify the URL we want to manipulate as the first parameter. As a second parameter, we define the instructions that cause the body to be manipulated. To do this, we first use route.fetch to store the actual response to queries about the page in a variable. We then change this answer by adding a “bad” script at the end. Using route.fulfill, we instruct Playwright to return the manipulated body when the page is accessed. After the method has been called in the test, every call to the page is intercepted by Playwright and the HTML body of the response is replaced by the manipulated body. If the script should be called due to an insufficient CSP, we also use Playwright’s route method. This redirects the call for the script to a script that we have defined:

async function setupRouteForEvilScript(page: Page) {
  await page.route("https://bad.test/evil.js", async (route) => {
    const jsContent = `console.log("Hello world!");`;
    await route.fulfill({
      status: 200,
      contentType: "application/javascript",
      body: jsContent,
    });
  });
}

If the page with the manipulated body is called up during the test execution, a warning is issued in the console of the browser and the “evil” script is not loaded.

The screenshot was taken during the test execution and in it you can see several violated CSP rules. These error messages are written to the array mentioned at the beginning. They are validated in a separate file, just as the CSP in the header of the HTML response. If the error message changes or does not appear at all during a test execution, the test will fail, and a cause and solution must be sought.

Prevent clickjacking with CSP

CSP can also be used to prevent “malicious” websites from embedding our page into their website using an iframe element, a so-called clickjacking attack. By embedding the website, our site is overlaid by the malicious website and neither the users nor we as the operator recognize that functions are unintentionally executed on the site. To prevent this, “frame-ancestors `none`” is added to the CSP. This will cause any embedding attempts to fail. For our test, we created a minimal website that includes an iframe element on our page. We used the route method again.

async function setupRouteForIframeSite(page: Page) {
  const body = `<!DOCTYPE html>
    <head>
      <meta charset="utf-8">
      <title>ClickJacking Test</title>
    </head>
    <body><iframe src="${page.url()}"</body>
    </html>`;
  await page.route("https://bad.test/clickjacking", (route) =>
    route.fulfill({
      contentType: "text/html;charset=utf-8",
      body,
    }),
  );
}

The method setupRouteForIframeSite works so that when the URL “https://bad.test/clickjacking” is called in the test, the page defined in the method is called. If the CSP is configured correctly, then the iframe element will not work. In addition, an error message is displayed on the page in the console.

This can be seen in the screenshot above. The error message also specifies the breached CSP “frame-ancestors `none`”. This error message is written to a validation file as described above and checked each time the test runs.

Test CSRF attack

Finally, we present a CSRF scenario that can be checked by means of end-to-end tests in Playwright. The first step is Playwright logging in to the software to be tested. For this test, we have created two minimal websites that send a query to our software to be tested when you click on a link. However, this is not obvious to a user at first glance. For demonstration purposes or test purposes, we used a state-changing GET request.

We test both a cross-origin and a same-site case.

The first website has a different domain from the page being tested. The second website has a subdomain of our page as a URL (as pictured above). As you can see, for the purposes of this test it has been kept very minimal and essentially only contains the malicious link. When Playwright clicks on the link in the test, we always check that an error message appears when calling up the link. In addition, we use Playwright’s route method to monitor the endpoint that is attacked by the malicious calls, in this case clicking on the link.

async function monitorAttackedEndpoint(
  page: Page,
) {
  await page.route(attackedEndpoint, async (route) => {
    const response = await route.fetch();
    expect(response.status()).toBe(403);

    await route.fulfill({ response: response });
  });
}

One of the methods of preventing such an attack is the use of CSRF cookies. This prevents the endpoint from responding to the malicious request, as the malicious site does not have access to the CSRF cookies that must be sent along for a successful request. An http-403 error code is returned in our software when an attempted CSRF attack occurs. We check this using the method presented above.

Final Reflections

In this article we used some examples to show how security aspects for web applications, including CSP or CSRF, can be tested automatically in conjunction with Playwright through end-to-end tests. It was shown how some different aspects can be tested, such as the presence of the expected CSP in the http response. The tests can be adapted to the needs of different web applications and thus can be used across projects. The tests presented are only a small excerpt of possible security tests that can be automated. Other aspects of security, such as access authorizations or brute force attacks, can also be tested automatically with the help of end-to-end tests by Playwright.