Playwright

The official Flakiness.io reporter for Playwright Test. The open source implementation is available at github.com/flakiness/playwright.

A custom Playwright test reporter that generates Flakiness Reports from your Playwright test runs. The reporter automatically converts Playwright test results into the standardized Flakiness JSON format, capturing test outcomes, attachments, system utilization, and environment information. This enables you to track test flakiness over time, analyze patterns across different environments, and identify regressions in your test suite.

Quick Start

Install the Flakiness.io Playwright reporter:
Terminal window
```
npm install @flakiness/playwright
```

Configure the reporter in your playwright.config.ts file:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['@flakiness/playwright']
  ],
});

Run your tests. The report will be automatically generated in the ./flakiness-report folder:
Terminal window
```
npx playwright test
```
View the interactive report:
Terminal window
```
npx flakiness show ./flakiness-report
```

CI/CD Integration

Reports are automatically uploaded to Flakiness.io at the end of each test run if a valid access token is provided. If no token is present, the report is still generated locally and can be uploaded manually later.

When the access token is provided, you’ll see a confirmation in your terminal:

  45 passed (1.5m)
[flakiness.io] ✓ Uploaded as https://flakiness.io/flakiness/flakiness/run/6574

To set up automatic uploads, you need your project’s Access Token. You can find this in your project settings on flakiness.io.

GitHub Actions

Add your Flakiness.io access token to your repository:

Navigate to your GitHub repository settings
Go to Secrets and Variables → Actions
Add a new secret named FLAKINESS_ACCESS_TOKEN
Set its value to the access token from your Flakiness.io project settings
Pass the secret as an environment variable to the npx playwright test step

# In your GitHub workflow file (.github/workflows/playwright.yml)
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      # ... other steps like checkout, setup Node.js, etc.
      - name: Run Playwright tests
        env:
          FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
        run: npx playwright test

Configuration

The reporter accepts the following configuration options:

`endpoint?: string`

Custom Flakiness.io endpoint URL for uploading reports. Defaults to the FLAKINESS_ENDPOINT environment variable, or https://flakiness.io if not set.

Use this option to point to a custom or self-hosted Flakiness.io instance.

reporter: [
  ['@flakiness/playwright', { endpoint: 'https://custom.flakiness.io' }]
]

`token?: string`

Access token for authenticating with Flakiness.io when uploading reports. Defaults to the FLAKINESS_ACCESS_TOKEN environment variable.

If no token is provided, the report will still be generated locally but won’t be uploaded automatically.

reporter: [
  ['@flakiness/playwright', { token: 'your-access-token' }]
]

`outputFolder?: string`

Directory path where the Flakiness report will be written. Defaults to flakiness-report in the current working directory, or the FLAKINESS_OUTPUT_DIR environment variable if set.

reporter: [
  ['@flakiness/playwright', { outputFolder: './test-results/flakiness' }]
]

`open?: 'always' | 'never' | 'on-failure'`

Controls when the report viewer should automatically open in your browser after test completion.

'on-failure' (default): Opens the report only if tests failed and running in an interactive terminal (not in CI)
'always': Always opens the report after test completion (when running in an interactive terminal)
'never': Never automatically opens the report

reporter: [
  ['@flakiness/playwright', { open: 'always' }]
]

`collectBrowserVersions?: boolean`

When enabled, the reporter will launch each browser type used in your Playwright projects to detect and record the actual browser version. This information is added to the environment metadata.

Note: This option requires launching browsers, which adds overhead to report generation. Enable only when browser version information is critical for your analysis.

reporter: [
  ['@flakiness/playwright', { collectBrowserVersions: true }]
]

Environment Variables

The reporter respects the following environment variables:

FLAKINESS_ACCESS_TOKEN: Access token for Flakiness.io uploads (equivalent to token option)
FLAKINESS_ENDPOINT: Custom Flakiness.io endpoint URL (equivalent to endpoint option)
FLAKINESS_OUTPUT_DIR: Output directory for reports (equivalent to outputFolder option)
FK_ENV_*: Custom environment variables prefixed with FK_ENV_ are automatically included in the environment’s metadata. The prefix is stripped and the key is converted to lowercase.

Example:

export FK_ENV_DEPLOYMENT=staging
export FK_ENV_REGION=us-east-1

This will result in the environment containing:

{
  "metadata": {
    "deployment": "staging",
    "region": "us-east-1"
  }
}

Flakiness.io platform will create a dedicated history for tests executed in each unique environment. This means tests run with FK_ENV_DEPLOYMENT=staging will have a separate timeline from tests run with FK_ENV_DEPLOYMENT=production, allowing you to track flakiness patterns specific to each deployment environment.

Example Configuration

Here’s a complete example with all options:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['@flakiness/playwright', {
      endpoint: process.env.FLAKINESS_ENDPOINT,
      token: process.env.FLAKINESS_ACCESS_TOKEN,
      outputFolder: './flakiness-report',
      open: 'on-failure',
      collectBrowserVersions: false,
    }]
  ],
  // ... rest of your config
});