Pytest

Demo

Report demo is available at https://flakiness.io/flakiness/pytest-flakiness

The official Flakiness.io reporter for pytest.

Installation

1. Install using uv (recommended):

   uv add --dev pytest-flakiness

   Or via standard pip:

   pip install pytest-flakiness

CI/CD Integration

To upload reports, you need your project’s Access Token. You can find this in your project settings on flakiness.io.

When the Access Token is provided, the reporter will upload the report to Flakiness.io in the end of the run. You will see a confirmation in your terminal:

PASSED [100%]
==============================
✅ [Flakiness] Report uploaded: https://flakiness.io/your_org/your_proj/run/1
==============================

GitHub Actions

Add your Flakiness.io access token to your repository:

1. Navigate to your GitHub repository settings
2. Go to Secrets and Variables → Actions
3. Add a new secret named FLAKINESS_ACCESS_TOKEN
4. Set its value to the access token from your Flakiness.io project settings
5. Pass the secret as an environment variable to the npx playwright test step

# In your GitHub workflow file (.github/workflows/tests.yml)
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      # ... other steps like checkout, setup uv, etc.
      - name: Run Pytest tests
        env:
          FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
        run: pytest

Usage

Once installed, simply run pytest. The reporter will automatically activate, aggregate test results, and create Flakiness Report in the flakiness-report directory.

pytest

The generated report can be viewed interactively via the Flakiness CLI Tool:

flakiness show

Tip

Make sure to add flakiness-report directory to your .gitignore:

flakiness-report/

Configuration Options

All options can be set via environment variables or command-line flags:

Flag	Environment Variable	Description
--flakiness-name	FLAKINESS_NAME	Name for this environment. Defaults to pytest
--flakiness-output-dir	FLAKINESS_OUTPUT_DIR	Local directory to save JSON report. Defaults to flakiness-report
--flakiness-access-token	FLAKINESS_ACCESS_TOKEN	Your Flakiness.io access token (required for upload)
--flakiness-endpoint	FLAKINESS_ENDPOINT	Flakiness.io service endpoint. Defaults to https://flakiness.io

Custom Environment Data

You can add custom metadata to your test runs using FK_ENV_* environment variables:

export FK_ENV_GPU_TYPE="H100"
export FK_ENV_DEPLOYMENT="staging"
pytest

The FK_ENV_ prefix is removed and keys are lowercased, e.g. FK_ENV_DEPLOYMENT becomes deployment, and FK_ENV_GPU_TYPE becomes gpu_type.

License & Source Code

The source code is available under MIT license at https://github.com/flakiness/pytest-flakiness