Skip to content
Flakiness.io

Reports

In Flakiness.io, a Report is a unified view of your test results across different timelines and commits.

You can generate reports covering multiple days, specific Pull Requests, or individual commits. To find exactly what you need, reports can be filtered by environment or narrowed down using FQL.

Aggregated Status

Section titled “Aggregated Status”

When viewing a report containing multiple tests (e.g., all tests from the last week), Flakiness.io calculates a single Aggregated Status.

This status is determined by the “worst” individual result found within the collection, ensuring that critical issues are prioritized. The aggregation follows this specific hierarchy:

  1. 🔥 Regressed At least one test has regressed.
  2. ❌ Failed No regressions found, but at least one test failed.
  3. ⚠️ Flaked The number of flaky tests exceeds your Acceptable Flakiness Ratio.
  4. ✅ Passed All tests passed, or the number of flaky tests is within the acceptable limit.
  5. ∅ Skipped All tests in the report were skipped.

Acceptable Flakiness Ratio

Section titled “Acceptable Flakiness Ratio”

The Acceptable Flakiness Ratio is a project-level setting that defines your tolerance for instability.

It acts as a threshold for when flakiness impacts the overall Aggregated Status. For example, if your ratio is set to 5%:

  • 3 flaky tests out of 100 total (3%) will result in a Passed status.
  • 6 flaky tests out of 100 total (6%) will result in a Flaked status.