Flakiness JSON

Open Source

Flakiness JSON specification is available at https://github.com/flakiness/flakiness-report

Flakiness JSON is a test report format designed specifically for Flakiness.io service. The format is feature-rich, and can be used with or without Flakiness.io service.

Distinct features of Flakiness Report are:

• Each report includes a source code revision.
• Each report contains detailed information about test execution environment.

Use Cases

Flakiness JSON is designed to be language and framework agnostic and supports wide variety of different test types:

1. Integration Tests: Flakiness JSON can encode complex environments and store system utilization data to help test debugging.
2. Unit Tests: Flakiness JSON is optimized to efficiently store tens of thousands of test results.
3. Performance Tests: Since every test result comes with a duration, Flakiness JSON can be used to report performance test results.

Creating Reports

Flakiness JSON is a JSON file that follows Flakiness JSON specification. Oftentimes, this JSON file is accompanied by a set of files - attachments. Attachments are placed next to the report.json file, in the attachments/ directory.

Flakiness JSON can be created by various integrations. It is also very easy to build an in-house report generators relying on the Flakiness JSON specification - read more in custom reporters.

Viewing

A generated Flakiness JSON can be viewed via the Flakiness.io CLI:

flakiness show ./flakiness-report

Clients can link local Git checkout to a project on Flakiness.io service. In this case, locally-generated report will be matched against cloud history of the project:

flakiness link https://flakiness.io/myorg/myproj

Flakiness.io service

Users can create a project on the Flakiness.io service to store & analyze their Flakiness JSONs in the cloud.

Some integrations can be configured to auto-upload Flakiness JSON from CI to the Flakiness.io service; others rely on the Flakiness CLI to upload reports manually.

Once Flakiness JSON is uploaded, it’s called a run; each run has a direct link to view report on-line.

Flakiness.io service will store & analyze all uploaded reports in a chronological order defined by the Git history.