This Month in Our Test Infra: March 2025 issue

Apr. 13, 2025 · Jieyou Xu on behalf of the Bootstrap Team

This Month in Our Test Infra: March 2025 issue

This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository1 between 2025-03-11 and 2025-04-13.

During this period, there are some significant changes to compiletest and bootstrap that contributors should be aware of.

As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!

Thanks to everyone who contributed to our test infra!

Highlights

compiletest now supports matching diagnostics on lines below

UI error annotations previously could match against diagnostics emitted for previous lines (//~^) or same line (//~), but couldn't express matching against diagnostics emitted for subsequent lines. Following #139100, test writers can now use //~v to match against diagnostics emitted for subsequent lines. This is particularly useful for e.g. parser tests where the test file may be exercising lexer/parser diagnostics and the test file itself is not syntactically valid (so can't use comments afterwards to match against previous lines).

Example:

// tests/ui/parser/issues/issue-103451.rs

struct R { }
//~vv ERROR this file contains an unclosed delimiter
struct S {
    x: [u8; R

compiletest now supports precise and exhaustive matching on diagnostics without line information associated with the main test file

Previously, test writers had to resort to //@ error-pattern to match against diagnostics which do not have associated line information in the main test file. Now, test writers may use //~? DIAGNOSTIC_KIND diagnostic_message to match against a diagnostic with diagnostic kind DIAGNOSTIC_KIND (e.g. ERROR or HELP) with message diagnostic_message. See compiletest: Support matching on diagnostics without a span #138865. //~? may appear on any line of the test, but are conventionally placed at the end.

Example:

// tests/ui/invalid-module-declaration/invalid-module-declaration.rs

mod auxiliary {
    mod foo;
}

fn main() {}

//~? ERROR file not found for module `baz`

Compared to //@ error-pattern, //~? error annotations are precise and exhaustive.

  • Precise: //~? annotations will not match against diagnostics that do have line information.
  • Exhaustive: if multiple diagnostics lacking line information in the main test file are present, then all of them need to be annotated with //~?.

Combined, //~? helps to prevent accidentally blessing away diagnostics that don't have associated line information on the main test file.

//@ error-pattern are still useful for e.g. runtime error messages that do not have specific spans, or compile time messages that require imprecise matching due to multiline platform-specific diagnostics. //~? should be preferred where suitable.

Notable changes

This section is intended to be like "compatibility notes" but for human test writers.

In preparation of stage 0 std redesign, it is now possible to use stage 0 libtest for compiletest

The stage 0 std redesign means that building the library tree would require a stage 1 compiler. compiletest is somewhat strange in that it currently depends on in-tree libtest -- meaning that with the stage 0 std redesign, changes to the compiler tree would necessitate rebuilding compiletest, causing more friction for compiler development cycle. To mitigate this, bootstrap now supports a new configuration option to switch compiletest to depend on stage 0 libtest instead of in-tree libtest, so that changes to compiler/ tree would not cause compiletest to rebuild.

[build]
compiletest-use-stage0-libtest = true

Note that as a trade-off, we do check this force-stage0-libtest configuration in PR CI. This means that there are possible windows of time where libtest may be modified before bumping stage 0 compiler, which may require the PR author modifying libtest to add cfg(bootstrap)/cfg(not(bootstrap)) to compiletest bits depending libtest programmatic API as suitable. This is done under the assumption that libtest programmatic API changes are comparatively much less frequent. If this proves to be a significant burden, the bootstrap team would be open to revisiting this scheme.

Note in over the long term, we'd like to migrate away from depending on in-tree libtest (both to render this hack unnecessary and to have more control over test execution).

bootstrap's config.toml has been renamed bootstrap.toml

Previously, bootstrap used config.toml as the configuration file under checkout root. This is potentially confusing since Cargo also uses config.toml as its configuration file name. Now, bootstrap instead uses bootstrap.toml as the configuration file name. The example config file config.example.toml has also been renamed to bootstrap.example.toml.

config.toml is temporarily accepted as fallback config file to mitigate migration churn across different checkouts, but will be phased out in the future.

compiletest now enforces stricter parsing of diagnostics kinds

Previously, compiletest was very lax in the casing of diagnostic kinds (i.e. the ERROR/HELP portion of error annotations). This mean that annotations such as

//~ Error xxx
//~ Error: xxx

were also accepted compared to the more common //~ ERROR or //~ error forms.

Eventually, we would like to canonicalize the error annotations into one form. To make this transition less abrupt, as an intermediate step, compiletest will now:

  • Enforce that error annotation diagnostic kinds must either be full caps (e.g. ERROR) or lowercase (e.g. error). Mixed cases like //~ Error will now be rejected.
  • Empty diagnostics (such as empty //~ NOTE) will no longer be silently ignored. These indicate silly empty diagnostic notes emitted by rustc which should be fixed.

compiletest now enforces that error annotations are required even if //@ error-pattern directives are used

Related to previous //~? improvements, compiletest will now also check for error annotations (//~) in tests that use //@ error-pattern to minimize risk of accidentally blessing away diagnostics, including diagnostics without associated line information for the main test file..

In exceptional cases, the test writer may opt out of the error annotation checks via //@ compile-flags: --error-format=human.

compiletest now allows opting in to non-exhaustive matching of a specific diagnostic kind via new directive //@ dont-require-annotations: DIAGNOSTIC_KIND

compiletest now has a new directive //@ dont-require-annotations: DIAGNOSTIC_KIND (where DIAGNOSTIC_KIND is e.g. ERROR/HELP/SUGGESTION) to allow opting into non-exhaustive matching for the specified DIAGNOSTIC_KIND. This includes the ERROR diagnostic kind, where it was not possible to opt-out of exhaustive matching previously.

This directive should be used with caution and sparingly.

compiletest no longer accepts {true, false} as revision names

In test suites that support revisions, previously //@ revisions: true and //@ revisions: false were accepted as revision names. However, this is confusing because they will be automatically registered as --cfg={true,false} to rustc, but the test writer would have to use cfg(r#true) and cfg(r#false) in the test. Hence, the {true,false} revision names are no longer permitted by compiletest.

compiletest now supports a //@ needs-crate-type directive

Test writers can now use a //@ needs-crate-type directive to guard test execution based on whether the target platform supports all of the required crate types.

The directive accepts a comma-delimited list of valid crate types that are accepted by the rustc --crate-type flag. E.g. //@ needs-crate-type: dylib, proc-macro.

Example:

// This test would be ignored on targets e.g. `wasm32-unknown-unknown` that
// do not (currently) support `dylib` crate type.

//@ needs-crate-type: dylib
//@ compile-flags: --crate-type=dylib

fn foo() {}

compiletest now trims whitespace from the env var name passed to //@ {unset,}-{exec,rustc}-env directives

compiletest has a long-standing quirk where

//@ rustc-env: RUSTC_BOOTSTRAP=1

was not the same as

//@ rustc-env:RUSTC_BOOTSTRAP=1

where the former is treated as ⌴RUSTC_BOOTSTRAP. The same was true for //@ exec-env, and the //@ unset-{exec,rustc}-env directives.

This has now been fixed. compiletest will now trim whitespace from the env var name, so that the aforementioned forms are now equivalent:

//@ rustc-env: RUSTC_BOOTSTRAP=1
//@ rustc-env:RUSTC_BOOTSTRAP=1

both correspond to env var name RUSTC_BOOTSTRAP.

compiletest now enforces that //@ edition directive must be used instead of //@ compile-flags: --edition=xxx

compiletest has an --edition flag that permits changing the default edition that tests are run with. However, no test suites currently pass with that set to non-2015-edition, but in the future we would like to be possible to run the test suites across all editions to ensure we have good test coverage over all editions (subject to further design concerns).

To make this remotely possible, we first need to ensure compiletest's --edition override flag functions. This means that now, compiletest will reject rustc --edition flags passed via //@ compile-flags in favor of the //@ edition directive, to ensure that compiletest can override the edition of the test.

If the test author needs to exercise the behavior of the --edition rustc flag itself, a run-make test should be used.

Upcoming changes

The following changes have not merged yet, but will merge in the near future or we are actively working towards them.

compiletest will soon also treat SUGGESTION error annotations as viral

Previously, if the test writer specifies //~ HELP or //~ NOTE, then compiletest treated them as viral -- if one is specified, then all HELP/NOTE diagnostics of the same diagnostic kind must be exhaustively specified. For consistency and to be more strict in checking what diagnostics the compiler emits, we plan to make //~ SUGGESTION annotations also viral.

This viral exhaustive-matching behavior can be opt-out through the newly introduced //@ dont-require-annotations: SUGGESTION (see the Notable changes section for the description of this new directive).

run-make tests (and the run-make-support library) will soon cross-compile for target platform by default

We're working on fixing the run-make test suite and the run-make-support library to properly test cross-compile target platform by default. Test authors would still be able to use //@ ignore-cross-compile if the test can only work on host.

See https://github.com/rust-lang/rust/pull/138066 and related work-in-progress efforts.

PR listing

Improvements

Fixes

Cleanups

Documentation updates

Note that since rustc-dev-guide became a josh subtree in rust-lang/rust, some doc updates are made alongside the rust-lang/rust PR themselves (which are not redundantly listed here).