This Month in Our Test Infra: October 2024

Nov. 4, 2024 · Jieyou Xu on behalf of the Bootstrap Team

This Month in Our Test Infra: October 2024

This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository1 for October 20242. It also includes brief descriptions of on-going work.

As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!

Thanks to everyone who contributed to our test infra!

Highlights

compiletest now supports bringing your own custom diff tool

compiletest (and bootstrap) now supports bringing your own custom diff tool (#131181).

A ui test failure showing stderr snapshot diff produced by a custom differ

This only affects the visual diff generation (i.e. maybe you like a different visual style). The .stderr snapshots and such are not affected.

If you want to use your favorite diff tool for generating the visual diffs, you can modify config.toml's build.compiletest-diff-tool option:

[build]
# What custom diff tool to use for displaying compiletest tests.
#compiletest-diff-tool = <none>

Thanks to @dev-ardi for the implementation!

minicore test auxiliary and //@ add-core-stubs directive

ui, assembly and codegen tests can now use the //@ add-core-stubs directive to conditionally build a minicore test auxiliary which provides core stubs (#130693). This is so that we can share core stubs between cross-compiling tests that only need to build for the cross-compile target and avoid having to reinvent and maintain duplicate minicore copies in each of such test3.

Previously, having to reinvent a minicore every time you want to add a distinct ui/assembly/codegen test (for checking e.g. cross-compile ABI) is a significant source of contributor friction and makes it more prone to introduce mistakes in the minicore copies. This is also particularly annoying for compiler contributors who want to introduce new lang items, as they found themselves having to update multiple copies of such core stubs.

Note that currently, the shared tests/auxiliary/minicore.rs test auxiliary is still quite minimal. The plan is to land the test infrastructure first, then we can incrementally add more core stubs to the shared test auxiliary.

Example usage:

// tests/ui/abi/my-abi-test.rs

//@ add-core-stubs
//@ compile-flags: --target x86_64-pc-windows-msvc
//@ needs-llvm-components: x86

#![crate_type = "lib"]
#![feature(no_core)]
#![no_std]
#![no_core]

extern crate minicore;
use minicore::*;

struct Meow;
impl Copy for Meow {} // `Copy` is provided by `minicore`!

See the context issue, MCP and tracking issue for more info on the original motivations. See the rustc-dev-guide chapter for example usage.

PR listing

Improvements

Fixes

Cleanups

Documentation updates

On-going efforts

Note: there are certainly many more spontaneous efforts, this is more what I know is "planned".

  1. The test infra here refers to the test harness compiletest and supporting components in our build system bootstrap. This test infra is used mainly by rustc and rustdoc. Other tools like cargo, miri or rustfmt maintain their own test infra. ↩

  2. specifically 2024-10-09 to 2024-11-04. Note that the previous issue incorrectly used October in the filename and thus URL, but is actually the September issue. ↩

  3. You can say we currently have more of a... "multicore" situation, heh. ↩

  4. This is part of T-spec efforts to associate tests with Reference rules. ↩

  5. If you want to see what unsuppressed Windows Errors Reporting looks like for the run-make test suite, see https://github.com/rust-lang/rust/issues/132092#issuecomment-2436615833. ↩