Flakes and retries

A test that fails once and passes on a rerun tells you nothing about your code and costs a full build to find out. Selective handles it inside the run.

Retry in the same run

When a test fails, Selective puts it straight back on the queue and the next free node runs it again. The rest of the suite keeps going.

• Pass on retry. The failure is removed from the run’s result, the run can still go green, and the event is recorded as a flake occurrence against that test.
• Fail on retry. The failure stands. The run is red for that test and the output from the attempt is in the dashboard and in selective tail.

Retrying on a different node also rules out one common cause: a node that was in a bad state.

Flake status

Every test in a suite has a status:

Status	Meaning
stable	No recent flake occurrences.
flaking	Has passed on retry recently. Shown on the flakes dashboard with its rate and trend.
quarantined	Its failures don’t fail the build.

The flakes dashboard at app.selective.ci/flakes lists flaking tests with how often they flake, how that is trending, and the recent occurrences. From a test’s detail view you can:

• See the context of each occurrence, including which tests ran before it on that node. Order-dependent flakes come from the tests that ran before them, so this is usually the first thing to look at.
• Quarantine the test, so the build stays green while someone fixes it. Quarantined tests still run and are still reported; they just don’t fail the run.
• Mark it fixed, which returns it to stable and starts tracking fresh.
• Copy a prompt describing the flake, its recent occurrences and their context, for a coding agent to investigate.

What counts as a flake

A flake is a pass on retry within the same run. A test that fails on both attempts is a failure, not a flake, even if it later passes on a rerun of the whole build. Flake rate is occurrences over runs in a window, so a test that flakes in one of every three runs shows as 33 percent.