Full setup checklist

This page is the complete path from nothing to a suite that selects tests, records its map every day and is warmed up before its test nodes start. It is written so a coding agent can follow it top to bottom in a checkout of the repository. llms.txt has the same content in one plain-text file.

What only a person can do

An agent can’t do these. Ask for them, and continue with the rest meanwhile.

1. Create one suite per test command at app.selective.ci. A repository with an RSpec suite and a Minitest suite is two suites, each with its own API key.
2. Add each API key to the CI provider’s secrets. Use SELECTIVE_API_KEY for a repository with one suite; with several, give each its own secret name and map it to SELECTIVE_API_KEY in the job that runs that suite. Never commit a key or print it.
3. Install the Selective GitHub App on the repository, from the organization’s Settings → GitHub page in the dashboard. Selection needs it when the test job has no git checkout (RWX’s default clone, for example), and semantic selection always needs it to read the diff. Some plans require it for every build.
4. Set each suite’s mode on its settings page: record first, then shadow, then select or live (see step 9).
5. For live mode, ask support to turn on semantic selection for the organization: [email protected].

What the agent does

1. Find the suites

Look for every test command the CI config runs: bundle exec rspec, bundle exec rake test, bundle exec ruby -Itest, bin/rails test. Each distinct command is one suite. Report the list so the person can create a suite and key for each.

2. Add the gem

bundle add selective-ruby-rspec      # RSpec
bundle add selective-ruby-minitest   # Minitest

Make sure Gemfile.lock lists the CI platform, or the runner fails with Unable to find executable for <platform>:

bundle lock --add-platform x86_64-linux

3. Make sure CI can build the tracer

Installing the gem builds the selective_tracer native extension, which recording needs. It requires make and a C compiler. Most Ruby CI images have them; slim images may not. Add build-essential (Debian and Ubuntu) or the equivalent to the image if the install log says the tracer couldn’t be built. Without it everything works except recording maps.

4. Wire the CI provider

Follow the page for the provider: GitHub Actions, CircleCI, Semaphore, Buildkite, RWX, any other provider.

5. Swap the test command

bundle exec rspec
bundle exec selective rspec
bundle exec rake test
bundle exec selective minitest

Framework flags and arguments are passed through unchanged.

6. Run in parallel

Run the same command on more nodes. Every node with the same run identifiers joins one queue; nothing is pre-split.

7. Add the daily map recording job

Selection needs a map of which files each test executes. Schedule a full run of each suite once a day on the default branch with SELECTIVE_TEST_MAP_RECORD=1. That run records a new map; pull request runs keep selecting. The suite’s re-record interval (25 hours) is the fallback when the daily job didn’t run. Each provider page has an example under Daily map recording.

8. Add the warm-up step

A warm-up tells Selective a build is coming, so it fetches the commit’s file tree, loads the map and prepares the semantic decision before the test nodes connect. It’s optional; without it selection takes longer to apply once nodes are up. Each provider page has an example under Warm-up. The warm-up must send the same commit, base commit and target branch the runner reports, or its work isn’t reused. See what each provider reports.

9. Roll out selection

1. record: full runs, and maps are recorded. Keep it until a map exists (the suite settings page shows it).
2. shadow: still full runs, but every run shows what selection would have skipped, and flags any failing test it would have skipped. Watch a few days of pull requests.
3. select (the test map alone) or live (the test map plus semantic selection, for pull requests).

10. Open the pull request

Open one pull request with the gem, the CI changes, the recording job and the warm-up. Say which CI files changed and which person-only steps above are still open.

What each provider reports

The warm-up’s sha, base_sha and target_branch must match what the runner reports, because a prepared decision is keyed by suite, commit and base. The runner uses base_sha when it has one, otherwise target_branch.

Provider	sha	base_sha	target_branch
GitHub Actions	GITHUB_SHA (on pull_request events, the merge commit)	pull_request.base.sha on pull requests, before on pushes, read with jq (present on GitHub-hosted Ubuntu runners)	GITHUB_BASE_REF (pull requests only)
CircleCI	CIRCLE_SHA1	none	the orb’s target_branch
Semaphore	SEMAPHORE_GIT_SHA	the left side of SEMAPHORE_GIT_COMMIT_RANGE	SEMAPHORE_GIT_BRANCH, on pull requests only
Buildkite	BUILDKITE_COMMIT	none	BUILDKITE_PULL_REQUEST_BASE_BRANCH
RWX	RWX_GIT_COMMIT_SHA	none	none

SELECTIVE_SHA, SELECTIVE_BASE_SHA and SELECTIVE_TARGET_BRANCH, when set, override all of these. If you set them for the runner, send the same values in the warm-up.

How to check it worked

• The run appears at app.selective.ci, with every node connected to one run, and selective tail in the repository streams its results.
• The runner prints a test selection line near the start of each node’s output, for example:
  • Test selection: running all tests (no test map yet). Recording a test map.
  • Test selection (shadow mode): running all 2687 tests; selection would have skipped 2411.
  • Test selection: running 240 of 25806 tests (25566 unaffected by 3 changed files).
  • Test selection: running all tests (Gemfile.lock affects every test).
• The map is recorded: after a recording run, the suite’s settings page shows the map’s age and test count, or the error if the recording failed. If the runner prints that the map is due but these runners can't record, fix step 3.
• The warm-up answers 202 with JSON such as {"mode": "shadow", "map_id": "…", "semantic": "queued"}. 401 means the key is wrong; 400 means sha or git_repo_full_name is missing.
• The daily job records: its runs print recording requested and Recording a test map..

If a step doesn’t check out, see troubleshooting.