Test Statuses

For any timeline, Flakiness.io computes test statuses for all runs, commits, and days.

These statuses help you quickly understand test health at different levels - from individual test executions to broader time periods.

Test Status for Run

Let’s say we have a test testNervousSquirrel which was executed as part of some run.

Usually Flakiness Report contains the test status, so Flakiness.io uses just this:

Test Status for Commit

As discussed previously, all runs in Flakiness.io are attributed with a CommitId. So let’s say we now have a few runs for the same commit that belong to the same timeline.

In this particular case, Flakiness.io deduces that test testNervousSquirrel is flaky in the Windows Timeline for the commit D, since the same test both passed and failed.

In general, a test status for a commit and timeline is defined like this:

Status	Description
∅ skipped	The test has only Skipped runs
✅ passed	The test has only Passed runs and possibly some Skipped runs
❌ failed	The test has only Failed runs and possibly some Skipped runs
⚠️ flaked	Either the test has a Flaked run, or 1+ Passed and 1+ Failed runs.

Test Status for Multiple Commits

Now let’s say we’re tasked with computing test status for a bunch of commits:

Flakiness.io computes status of a test for given commits as its last known commit result.

In case of Windows timeline across these 4 commits, the test testNervousSquirrel will be marked as “passed”.

Test Status for Day

While source code evolves with commits, it is usually more convenient to perceive test history in terms of days.

Flakiness.io uses user’s timezone to split commit history into sub-histories per each day, and computes test status for each of these histories:

In our hypothetical example, we see that testNervousSquirrel:

❌ Ended the day of May 7, 2025 failing
⚠️ Was flaking on May 25, 2025
✅ And so far has perfect record Today, on May 26, 2025.