Test selection

Test selection runs only the tests that could be affected by a change. It is based on a test map: a record of which files each test executes, built by tracing real runs of your suite. In live mode Selective also reads the pull request’s diff and decides which tests it could affect (the semantic decision). Nothing about it lives in your project; the runner takes part automatically once selection is enabled for the suite.

Recording and selecting

There are two kinds of run.

Recording runs are full runs that trace every test and upload the set of files each one executed. A daily scheduled job with SELECTIVE_TEST_MAP_RECORD=1 is the way to keep the map fresh (see keeping the map fresh). If the map gets older than the suite’s re-record interval anyway, the next run records one.

Selecting runs send the git blob hash of every tracked file along with the test manifest. Selective compares them against the map and skips every test none of whose files changed. A test that has no entry in the map runs. A test whose files did change runs.

Suite modes

Selection is configured per suite in the dashboard. Move through the modes in order:

Mode	What happens
off	Every run is a full run. The default for a new suite.
record	Full runs, and Selective records maps on schedule. Use this first so it can learn the suite.
shadow	Full runs. Selective works out what it would have skipped and checks that against the run’s real failures, so you can trust the selection before turning it on.
select	Tests that the test map says can’t be affected by the change are skipped.
live	Tests are skipped on the test map and the semantic decision together. See the semantic decision.

Shadow mode never holds the build

In shadow mode selection runs alongside the build, not in front of it. Every test is queued for the nodes as soon as the manifest arrives, and the semantic evaluation runs in the background after the run finishes. The build takes as long as it would with selection off, and nothing about its result depends on how long the evaluation takes or whether it succeeds. Each run’s page shows what selection would have kept, and marks any failing test that it would have skipped.

The semantic decision

The test map knows which files each test executed, which is exact for code and blind to everything a trace can’t see: a changed fixture, a YAML file read at runtime, a migration. For pull requests, live and shadow mode add a second signal. Selective reads the diff, a reasoning model writes a brief of what the change affects, and a classifier scores every test file against that brief. A test runs when either the map or the brief says it might be affected.

The semantic decision needs the Selective GitHub App installed on the repository, so it can read the diff, and it is enabled per organization: contact support to turn it on. It only applies to changes off the default branch: in live mode, a run of the default branch runs everything.

How live mode applies it

The brief takes tens of seconds to write, and live mode doesn’t make the run wait for it before starting:

1. Tests the test map selects are queued as soon as the manifest arrives.
2. Tests the map would skip are held until the semantic decision arrives. Held tests it keeps are queued; the rest are dropped from the run.
3. If the decision fails, or isn’t ready within the deadline (150 seconds), every held test runs.

A failed or slow decision costs time and never skips a test. To take that wait out of the build, use a warm-up.

Warm-up

A warm-up tells Selective that a run is coming, so it can do the selection work before the test nodes connect. It is optional. Without it, selection works exactly the same; the selection just takes a few extra seconds to apply once the nodes are up.

What a warm-up saves depends on the mode:

Mode	Without a warm-up	With a warm-up
select	The first tests wait while Selective fetches the commit’s file tree and loads the test map, usually under a second and a few seconds when nothing is cached.	Both are ready when the nodes connect.
live	The same, and the tests the map would skip are held while the semantic decision is made, typically 20 to 45 seconds.	The decision is usually made by the time the nodes connect, so nothing is held.
shadow	The first tests wait on the file tree and map, as in select. The build never waits on the evaluation.	The file tree and map are ready, and the evaluation reuses the decision made during the warm-up rather than computing it again.

Send it as early as possible in the pipeline, before dependencies install, from a step the tests don’t depend on. Call it once per suite, with that suite’s API key:

curl -sS --max-time 20 -X POST https://app.selective.ci/warmup \
  -H "authorization: $SELECTIVE_API_KEY" \
  -H "content-type: application/json" \
  -d '{"sha": "'"$HEAD_SHA"'", "git_repo_full_name": "acme/shop", "base_sha": "'"$BASE_SHA"'", "branch": "'"$BRANCH"'"}' \
  || true

Field	Required	What it is
sha	Yes	The commit being built.
git_repo_full_name	Yes	owner/repo.
base_sha	No	The commit the change is compared against.
target_branch	No	The branch the change merges into.
branch	No	The branch being built.
language, framework, runtime_version	No	For example ruby, minitest, 3.3.6. Only needed when a suite has test maps for more than one environment; otherwise the suite’s newest map is used.

Send the same commit, base commit and target branch the runner will report. A decision is prepared for one exact change, and a run whose commit or base differs from the warm-up’s makes its own. What the runner reports depends on the provider; this table lists it, and each provider page has a ready-made warm-up: GitHub Actions, CircleCI, Semaphore, Buildkite, RWX, any other provider.

Selective answers 202 straight away with what it started, for example {"mode": "live", "map_id": "…", "semantic": "queued"}. semantic is queued when it began preparing a decision, already_prepared for a repeat notice about the same change, and not_applicable or off when the semantic decision doesn’t apply to this run or suite. A warm-up that fails or never arrives changes nothing, so don’t let its exit status fail the build.

Suite settings

Setting	Default	Meaning
Full-run patterns		Paths that force a full run when they change. Gemfile.lock, database schema, CI config and global test setup are the usual candidates.
Ignore patterns		Directories whose code isn’t the project’s own, such as vendored or generated code, so changes there don’t invalidate tests. Installed gems are ignored automatically.
Re-record interval	25 hours	Once the map is this old, the next run becomes a full run that records a new one. With a daily recording job it only fires when that job didn’t run.

Keeping the map fresh

Schedule a full run of the suite once a day on your default branch, with SELECTIVE_TEST_MAP_RECORD=1. That run records a new map and resets the clock, so pull request runs keep selecting and never pay for a recording.

The re-record interval (25 hours by default) is the fallback. If the daily job didn’t run, for example because it failed to start or only runs on weekdays, the next run after the interval becomes a full run that records a new map. Only one run records at a time; other runs keep selecting with the current map meanwhile.

An older map is never unsafe. Selective compares each file against the version the map recorded, not against the pull request’s base, so everything merged since the recording counts as changed. An old map just selects more tests, until it stops saving time.

Each provider page has a ready-made daily job: GitHub Actions, CircleCI, Semaphore, Buildkite, RWX. Two examples:

On GitHub Actions, add a schedule to the workflow and set the variable only for scheduled runs:

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "0 6 * * *"

jobs:
  test:
    env:
      SELECTIVE_TEST_MAP_RECORD: ${{ github.event_name == 'schedule' && '1' || '' }}

On RWX, add a cron trigger that passes an init value, and set the variable from it:

on:
  github:
    pull_request:
      init:
        record-map: false
  cron:
    - key: selective-map-recording
      schedule: "0 6 * * *"
      init:
        record-map: true

tasks:
  - key: tests
    run: bundle exec selective rspec
    env:
      SELECTIVE_TEST_MAP_RECORD: ${{ init.record-map }}

The suite’s settings page shows how old each map is, a recording in progress, and the error from the last recording if it failed. Runners need the native tracer to record (see below); when they don’t have it, a due map isn’t re-recorded and runs keep selecting with the old one, and both the run and the settings page say so.

Environment variables

Variable	Effect
SELECTIVE_TEST_MAP_RECORD=1	Record a map on this run, however fresh the current one is.
SELECTIVE_TEST_MAP_DISABLE=1	Take no part in selection on this run: run everything, record nothing.
SELECTIVE_TEST_MAP_IGNORE=dir1,dir2	Extra directories to ignore on this run, on top of the suite’s ignore patterns.

Recording needs the tracer

Recording uses the selective_tracer native extension built when the gem is installed. If your CI image has no C compiler, the gem still installs and selecting runs still work, but recording runs can’t produce a map. Either add build-essential (or the equivalent) to the image, or record maps from a job that has it.

What to expect

Selection is conservative by design. It never skips a test it has no information about, a change to a file no test was seen using runs everything, and so does any change to a full-run pattern. The saving on a given run depends on how localized the change is: a small change to one file typically runs a small fraction of the suite, a change to a widely shared module runs most of it.