Manual test selection

Set up manual test selection

The Tests Buildkite plugin runs a selection command that you provide before the step's command, and passes the tests that the command prints to bktec.

1. Create a .buildkite/select-tests.sh script that prints the tests to run, one selector per line. For most test runners, a selector is the test file path. The following example prints the RSpec spec files changed on the current branch. Replace the git diff command with your own selection logic:

   #!/usr/bin/env bash
   set -euo pipefail
   
   base_branch="${BUILDKITE_PULL_REQUEST_BASE_BRANCH:-main}"
   git fetch origin "${base_branch}" >&2
   
   # Print the spec files that were added or changed on this branch
   git diff --name-only --diff-filter=d --relative \
     "origin/${base_branch}...HEAD" -- '*_spec.rb'
   

   The plugin treats each line that the command prints to standard output as a selector, so send any other output to standard error. Make the script executable with chmod +x .buildkite/select-tests.sh.

   List each selector as bktec discovers it. Selectors are relative to the directory that bktec runs in, without a location prefix. If bktec runs in a subdirectory, such as backend in a monorepo, replace --relative with --relative=backend.

2. Add a step to your pipeline.yml file that runs bktec run with the Tests Buildkite plugin, and set manual-selection-command to the script:

   steps:
     - label: "Run selected tests"
       command: "bktec run"
       parallelism: 10
       plugins:
         - tests#v1.1.0:
             test-runner: rspec
             result-path: tmp/rspec-result.json
             manual-selection-command: ".buildkite/select-tests.sh"
   

   Setting manual-selection-command sets the selection strategy to manual, so the step doesn't need any selection flags.

Each parallel job runs the selection command. The jobs in a step share one test plan, so the command must print the same list in every job. If the command fails, the job fails. If the command prints no tests, the jobs pass without running any tests.

If none of the listed selectors match a test that bktec discovers, bktec run and bktec plan fail. To pass the job even when nothing matches, set fail-on-no-tests: false in the plugin configuration. When fewer tests are selected than there are parallel jobs, the remaining jobs exit without running tests. To size the step to the selected tests, use dynamic parallelism.

Use manual selection with dynamic parallelism

To size the test step to the selected tests, run the selection command on a planning step that runs bktec plan, and set a maximum parallelism and target time. bktec creates the test plan, then uploads the test step with the parallelism needed to reach the target time. The selection command only runs once, on the planning step. Learn more in Dynamic parallelism.

steps:
  - label: "Plan selected tests"
    key: "plan-selected-tests"
    command: "bktec plan --pipeline-upload .buildkite/selected-tests-template.yml"
    plugins:
      - tests#v1.1.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          max-parallelism: 10
          target-time: 2m
          manual-selection-command: ".buildkite/select-tests.sh"

The pipeline template runs the plan that bktec plan created. The test step doesn't need manual-selection-command, because the plan already contains the selected tests:

steps:
  - label: "Run selected tests"
    command: "bktec run"
    depends_on: "plan-selected-tests"
    parallelism: ${BUILDKITE_TEST_ENGINE_PARALLELISM}
    plugins:
      - tests#v1.1.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          plan-identifier: ${BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER}

If the selection command prints no tests, bktec plan doesn't upload the test step.

Run selected tests in Docker

The plugin runs the selection command on the agent before the step's command, and passes the selected tests to bktec in the BUILDKITE_TEST_ENGINE_SELECTION_SELECTORS environment variable. When bktec runs inside a container using the Docker plugin, add this variable to the Docker plugin's environment attribute. The Docker plugin's propagate-environment option doesn't pass it to the container. Without this variable, bktec sends no selectors and the job fails.

steps:
  - label: "Run selected tests"
    command: "bktec run"
    parallelism: 10
    plugins:
      - tests#v1.1.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          client-os: linux
          manual-selection-command: ".buildkite/select-tests.sh"
      - docker#v5.13.0:
          image: "ruby:3.4"
          expand-volume-vars: true
          volumes:
            - "$$BUILDKITE_TEST_ENGINE_CLIENT_PATH:/usr/local/bin/bktec"
          propagate-environment: true
          environment:
            - BUILDKITE_TEST_ENGINE_SELECTION_SELECTORS

Check the selection in the job log

bktec prints a planning summary at the start of each job, showing the selection this job requested and the selection that Test Engine applied:

Requested
  Selection strategy: manual
    selectors = <3 nonblank entries; 92 bytes>
  Parallelism: 10 (fixed)

Selection summary
  Applied strategy: manual
  Selected: 3 of 412 test selectors (0.7%)
  Estimated compute: 72.4s of 2304s (3.1%)
  Candidate timing coverage: 96%

Test Engine caches each test plan, so retrying a job reuses the original selection, even if the list has changed. The summary then shows Using existing plan. Start a new build to select tests again.

Git metadata

When a selection strategy is set, bktec also sends git metadata with the test plan request, including the full git diff against the base branch, which contains your source changes. Manual selection doesn't need this metadata. To turn it off, set BUILDKITE_TEST_ENGINE_COLLECT_GIT_METADATA to false in the pipeline-level env.

Review selection in Orchestration

Each build page's Tests tab includes an Orchestration page, which shows how bktec selected and split the tests in each test plan used in the build. To open it, select the build's Tests tab, then select Orchestration. A test plan appears once a job using it finishes.

The estimated time saved rows on the Test selection card only appear when at least half of the candidates have timing history. Estimated test wall-clock time saved only appears for steps with a fixed parallelism.

When at least one test was selected, the panel also shows a Selection map of the suite's test files, with the selected files in purple. Plans that mix test files or selectors with individual tests show Coverage map unavailable for plans with mixed test formats instead.