This Month in Our Test Infra: September 2024

Oct. 10, 2024 · Jieyou Xu on behalf of the Bootstrap Team

This Month in Our Test Infra: September 2024

This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository1 for September 2024. It also includes brief descriptions of on-going work.

As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!

Changes

run-make test suite now has access to a properly-staged cargo

bootstrap now builds a properly-staged cargo and makes it available for run-make tests. Previously, run-make tests just used whatever initial cargo bootstrap had access to, but this caused problems if a run-make test uses a cargo feature that's present in nightly but not in beta. We encountered some interesting build cache invalidation issues related to differing RUSTFLAGS in the process, but were able to fix them. We want to add mechanisms to bootstrap to make it harder to misuse RUSTFLAGS which may lead to hard-to-diagnose build cache invalidation in the future.

Why -Zon-broken-pipe=kill is required when building rustc was its own entire rabbit hole, in case you were curious.

More run-make migrations and fixes

The emit-to-stdout run-make test was ported to rmake.rs, only [10 more to go]. The remaining ones are stuck on being quite tricky. See the tracking issue for why we are transitioning away from Makefiles in run-make tests.

Misc:

run_make_support library updates

run_make_support is the support library built and made available to run-make tests.

compiletest improvements and fixes

We dropped compiletest's legacy directive check (e.g. // ignore-test hello no longer warns). This was originally added when we migrated from // to //@ to help test writers notice the new directives, but now a long time has passed so we can remove it as it was causing friction in adding new directives and authoring tests. For example, the Specification Team wanted to add a //@ reference directive, but the legacy directive check would trigger on:

// So what if we added a
// reference to the           <- `reference` is a known directive, and
//                               `// reference` looks suspcious!
// rustc-dev-guide?

This was added to initially to help migration from // to //@, but since a long time has passed we no longer need this check to help contributors know that legacy directives are being phased out.

We updated some compiletest normalizations and directive renaming. In particular, we restricted //@ ignore-mode-* directives to not accept all test suites, and later converted //@ ignore-mode-coverage-map and //@ ignore-mode-coverage-run to //@ ignore-coverage-map and //@ ignore-coverage-run because only coverage-map and coverage-run were special in that the same test source files ran under two test suite configurations.

We broke up compiletest's runtest.rs as it was previously massive, clocking in at 4710 lines. It's now around 2700 lines, so still massive, but at least slightly less so.

We added a help message upon crashes test failure that you can set COMPILETEST_VERBOSE_CRASHES=1 to get compiler stderr/stdout output from trying to build the failing crashes test.

We also registered tool docs for compiletest. There currently isn't much doc comments in compiletest, but having them getting built and made available as part of nightly rustc docs is a good first step.

Misc:

Testing documentation improvements

We improved testing docs in rustc-dev-guide. We added a testing best practices chapter, and updated the compiletest directives listing.

There's still a lot of room for improvement in our testing docs -- in compiletest, bootstrap and rustc-dev-guide, but one step at a time.

Test suite cleanups

We deleted an entire test suite run-pass-valgrind because it was never properly wired up and properly implemented, and was not used. It turns out deleting the test suite actually fixes a bug from 2017 run-pass-valgrind tests don't actually run in valgrind #44816 because you can't have a test suite related bug if the test suite doesn't exist!

Misc:

On-going developments

  • Add test infrastructure support for a minicore test auxiliary, so that #![no_std] cross-compiling build-only tests don't need to reinvent and reimplement core prelude stubs again and again. See https://github.com/rust-lang/rust/issues/131485.
  • We want to make RUSTFLAGS harder to misuse that can lead to tool build cache invalidation, leading to unnecessary rebuilds.
  • There's on-going effort to redesign stage0 std, to help make bootstrap staging more consistent and more intuitive.

  1. The test infra here refers to the test harness compiletest and supporting components in our build system bootstrap. This test infra is used mainly by rustc and rustdoc. Other tools like cargo, miri or rustfmt maintain their own test infra. ↩