This Month in Our Test Infra: September 2024
This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository1 for September 2024. It also includes brief descriptions of on-going work.
As usual, if you encounter bugs or UX issues when using our test infrastrucutre, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!
Changes
run-make
test suite now has access to a properly-staged cargo
bootstrap now builds a properly-staged cargo and makes it available for
run-make
tests. Previously, run-make
tests just used whatever initial cargo
bootstrap had access to, but this caused problems if a run-make
test uses a
cargo feature that's present in nightly but not in beta. We encountered some
interesting build cache invalidation issues related to differing RUSTFLAGS
in
the process, but were able to fix them. We want to add mechanisms to bootstrap
to make it harder to misuse RUSTFLAGS
which may lead to hard-to-diagnose build
cache invalidation in the future.
- bootstrap: Fix cargo staging for run-make tests #130739
- bootstrap/compiletest: Pass the current cargo to run-make tests #130642
- bootstrap: Prevent building cargo from invalidating build cache of other
tools due to conditionally applied
-Zon-broken-pipe=kill
via trackedRUSTFLAGS
#131155
Why -Zon-broken-pipe=kill
is required when building rustc was its own entire
rabbit hole, in case you were
curious.
run-make
migrations and fixes
More The emit-to-stdout
run-make
test was ported to rmake.rs
, only [10 more to
go]. The remaining ones are stuck on being quite tricky. See the tracking
issue for why we are transitioning away from Makefile
s in
run-make
tests.
- run-make: port
emit-to-stdout
to rmake.rs #131355
Misc:
run_make_support
library updates
run_make_support
is the support library built and made available to run-make
tests.
- run_make_support: Add
llvm-pdbutil
wrapper #130048 - run_make_support: Rename
Command::stdin
tostdin_buf
and addstd{in,out,err}
config helpers #129973 - run_make_support: Rectify symlink handling #130427
compiletest improvements and fixes
We dropped compiletest's legacy directive check (e.g. // ignore-test hello
no longer warns). This was originally added when we migrated from //
to //@
to help test writers notice the new directives, but now a long time has passed
so we can remove it as it was causing friction in adding new directives and
authoring tests. For example, the Specification Team wanted to add a
//@ reference
directive, but the legacy directive check would trigger on:
// So what if we added a
// reference to the <- `reference` is a known directive, and
// `// reference` looks suspcious!
// rustc-dev-guide?
This was added to initially to help migration from //
to //@
, but since a
long time has passed we no longer need this check to help contributors know that
legacy directives are being phased out.
- compiletest: Drop compiletest legacy directive check #131392
We updated some compiletest
normalizations and directive renaming. In
particular, we restricted //@ ignore-mode-*
directives to not accept all
test suites, and later converted //@ ignore-mode-coverage-map
and //@ ignore-mode-coverage-run
to //@ ignore-coverage-map
and //@ ignore-coverage-run
because only coverage-map
and coverage-run
were special
in that the same test source files ran under two test suite configurations.
- compiletest: Add
{{rust-src-base}}
(for sysroot src base) #129687 - compiletest: Restrict
ignore-mode-*
directives #131346 - compiletest: Simplify the compiletest directives for ignoring coverage-test modes #131400
We broke up compiletest's runtest.rs
as it was previously
massive, clocking in at 4710 lines. It's now around 2700 lines,
so still massive, but at least slightly less so.
We added a help message upon crashes
test failure that you can set
COMPILETEST_VERBOSE_CRASHES=1
to get compiler stderr/stdout output from trying
to build the failing crashes
test.
We also registered tool docs for compiletest. There currently isn't much doc comments in compiletest, but having them getting built and made available as part of nightly rustc docs is a good first step.
- bootstrap: Register tool docs for compiletest #130567
Misc:
- compiletest: Rename "runtest/crash.rs" to "runtest/crashes.rs" to be in line with the test directory #130973
Testing documentation improvements
We improved testing docs in rustc-dev-guide. We added a testing best practices chapter, and updated the compiletest directives listing.
- rustc-dev-guide: Revise testing chapters excluding the directives chapter #2088
- rustc-dev-guide: Revise directives docs #2089
- rustc-dev-guide: Revise test naming advice to discourage using issue numbers alone #2090
- rustc-dev-guide: Document compiletest directives
ignore-coverage-map
andignore-coverage-run
#2094 - rustc-dev-guide: Small follow-up to my "internal #[rustc_*] TEST attributes" PR #2082
- rustc-dev-guide: Note lldb debuginfo requires
python310.dll
to be present inPATH
on Windows #2076 - rustc-dev-guide: Document crashes test suite #2075
- rustc-dev-guide: Purge
run-pass-valgrind mentions
#2091 - rustc-dev-guide: Add documentation for
{{rust-src-base}}
#2079
There's still a lot of room for improvement in our testing docs -- in compiletest, bootstrap and rustc-dev-guide, but one step at a time.
Test suite cleanups
We deleted an entire test suite run-pass-valgrind
because it was never
properly wired up and properly implemented, and was not used. It turns out
deleting the test suite actually fixes a bug from 2017 run-pass-valgrind tests
don't actually run in valgrind
#44816 because you can't have a
test suite related bug if the test suite doesn't exist!
- bootstrap/compiletest/opt-dist: Remove valgrind test suite and support from compiletest, bootstrap and opt-dist #131351
Misc:
On-going developments
- Add test infrastructure support for a
minicore
test auxiliary, so that#![no_std]
cross-compiling build-only tests don't need to reinvent and reimplementcore
prelude stubs again and again. See https://github.com/rust-lang/rust/issues/131485. - We want to make
RUSTFLAGS
harder to misuse that can lead to tool build cache invalidation, leading to unnecessary rebuilds. - There's on-going effort to redesign stage0 std, to help make bootstrap staging more consistent and more intuitive.
-
The test infra here refers to the test harness compiletest and supporting components in our build system bootstrap. This test infra is used mainly by rustc and rustdoc. Other tools like cargo, miri or rustfmt maintain their own test infra. ↩