This Month in Our Test Infra: October 2024
This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository1 for October 20242. It also includes brief descriptions of on-going work.
As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!
Thanks to everyone who contributed to our test infra!
Highlights
compiletest
now supports bringing your own custom diff tool
compiletest
(and bootstrap) now supports bringing your own custom diff tool
(#131181).
This only affects the visual diff generation (i.e. maybe you like a different visual style). The
.stderr
snapshots and such are not affected.
If you want to use your favorite diff tool for generating the visual diffs, you can modify
config.toml
's build.compiletest-diff-tool
option:
[build]
# What custom diff tool to use for displaying compiletest tests.
#compiletest-diff-tool = <none>
Thanks to @dev-ardi
for the implementation!
minicore
test auxiliary and //@ add-core-stubs
directive
ui
, assembly
and codegen
tests can now use the //@ add-core-stubs
directive to
conditionally build a minicore
test auxiliary which provides core
stubs
(#130693). This is so that we can share core
stubs
between cross-compiling tests that only need to build for the cross-compile target and avoid
having to reinvent and maintain duplicate minicore
copies in each of such test3.
Previously, having to reinvent a minicore
every time you want to add a distinct
ui
/assembly
/codegen
test (for checking e.g. cross-compile ABI) is a significant source of
contributor friction and makes it more prone to introduce mistakes in the minicore
copies. This is
also particularly annoying for compiler contributors who want to introduce new lang items, as they
found themselves having to update multiple copies of such core
stubs.
Note that currently, the shared tests/auxiliary/minicore.rs
test auxiliary is still
quite minimal. The plan is to land the test infrastructure first, then we can incrementally add more
core
stubs to the shared test auxiliary.
Example usage:
// tests/ui/abi/my-abi-test.rs
//@ add-core-stubs
//@ compile-flags: --target x86_64-pc-windows-msvc
//@ needs-llvm-components: x86
#![crate_type = "lib"]
#![feature(no_core)]
#![no_std]
#![no_core]
extern crate minicore;
use minicore::*;
struct Meow;
impl Copy for Meow {} // `Copy` is provided by `minicore`!
See the context issue, MCP and tracking issue for more info on the original motivations. See the rustc-dev-guide chapter for example usage.
PR listing
Improvements
- General test infra: Add
minicore
test auxiliary and support//@ add-core-stubs
directive inui
/assembly
/codegen
tests #130693 - compiletest: Add test infra to explicitly test rustc with
autodiff
/enzyme
disabled #131470 - compiletest: Special case error message for a
build-fail
test that failed check build #131642 - compiletest: Document various parts of compiletest's lib.rs #131679
- compiletest: Fix unnecessary nesting in run-make test output directories #131764
- compiletest: Warn on redundant --cfg directive when revisions are used #131925
- compiletest: Disambiguate html-tidy from rust tidy tool #131941
- compiletest: Custom differ #131181
- compiletest: Don't allow test revisions that conflict with built in
cfg
s #131930 - compiletest: Dynamically link run-make support #132225
- compiletest: Improve robustness of LLVM version handling #132315
- compiletest: Add "reference" as a known compiletest header #1313824
tests/run-make
, CI: Addaarch64-gnu-debug
job #131207- meta: Tag PRs affecting compiletest with A-compiletest #131682
Fixes
- compiletest: Fix up-to-date checking for run-make tests #131681
- compiletest: Suppress Windows Error Reporting (WER) for run-make tests5
- compiletest: Error on trying to use revisions in run-make tests #131614
tests/run-make
, CI: Run the full stage 2run-make
test suite inx86_64-gnu-debug
#131917- bootstrap,
tests/run-make
: Don't stage-off to previous compiler when CI rustc is available #132006 - bootstrap,
tests/mir-opt
: Match stdRUSTFLAGS
for host and target formir-opt
test suite to fix double std build/rebuilds #131442 - emscripten: Fix bootstrap and compiletest handling of emscripten target tests as part of Fix most ui tests on emscripten target #131705
Cleanups
- compiletest: Extract auxiliary-crate properties to their own module/struct #131541
- compiletest: Rename directive
needs-profiler-support
toneeds-profiler-runtime
#131429 - compiletest: Move debugger setup code out of lib.rs #131638
- compiletest: Remove the one thing that was checking a directive's original_line #131585
- compiletest: Store test collection context/state in two structs #131870
- compiletest: Tidy up how tidy and tidy (html version) are disambiguated #131961
- compiletest: Make
line_directive
return aDirectiveLine
#132033 - compiletest: Rename
command-list.rs
todirective-list.rs
#132313 - compiletest: Remove the magic hacks for finding output with
lto=thin
#131524 - compiletest: Simplify the choice of
--emit
mode for assembly tests #131525 - compiletest: Move debugger setup code out of lib.rs #131638
Documentation updates
- rustc-dev-guide: Document compiletest directives
ignore-coverage-map
andignore-coverage-run
#2094 - rustc-dev-guide: Rename
needs-profiler-support
toneeds-profiler-runtime
#2095 - rustc-dev-guide: Fix and update docs for
needs-force-clang-based-tests
#2085 - rustc-dev-guide: Add redirects for integration-testing and headers #2092
- rustc-dev-guide: Describe minicore test auxiliary and directive #2097
- rustc-dev-guide: Fix minicore.rs link #2122
- rustc-dev-guide: Add a link for the
reference
compiletest header #2096
On-going efforts
Note: there are certainly many more spontaneous efforts, this is more what I know is "planned".
-
The test infra here refers to the test harness compiletest and supporting components in our build system bootstrap. This test infra is used mainly by rustc and rustdoc. Other tools like cargo, miri or rustfmt maintain their own test infra. ↩
-
specifically 2024-10-09 to 2024-11-04. Note that the previous issue incorrectly used October in the filename and thus URL, but is actually the September issue. ↩
-
You can say we currently have more of a... "multicore" situation, heh. ↩
-
This is part of T-spec efforts to associate tests with Reference rules. ↩
-
If you want to see what unsuppressed Windows Errors Reporting looks like for the
run-make
test suite, see https://github.com/rust-lang/rust/issues/132092#issuecomment-2436615833. ↩