This Month in Our Test Infra: December 2024

Jan. 10, 2025 · Jieyou Xu on behalf of the Bootstrap Team

This Month in Our Test Infra: December 2024

Happy new year, dear reader! This is the last This Month in Our Test Infra issue for 2024.

This is a quick summary of the changes in the test infrastructure for the rust-lang/rust repository1 for December 2024.

As usual, if you encounter bugs or UX issues when using our test infrastructure, please file an issue. Bugs and papercuts can't be fixed if we don't know about them!

Thanks to everyone who contributed to our test infra!

Highlights

rustc-dev-guide is now a josh subtree!

Previously, rustc-dev-guide was a submodule inside rust-lang/rust, and updates to rustc-dev-guide had to be done in a separate PR against the rustc-dev-guide repository.

Now, thanks to @Kobzol's efforts (which included overcoming many unforeseen obstacles), rustc-dev-guide is now a josh subtree. This is a significant improvement for contribution workflows because it means that documentation updates to rustc-dev-guide can accompany the implementation change in rust-lang/rust in the same PR. The reduction in contribution friction also encourages rustc-dev-guide updates because you no longer have to author and maintain two separate PRs.

compiletest will now show the difference between normalized output and actual output for differing lines

Previously, it can be very difficult to tell when a ui test fails what the actual output is pre-normalization, as you would have to resort to --nocapture or similar.

Now, compiletest will also show the pre-normalization mismatched output lines on failure to make this easier to tell. Example output:

failures:

---- [ui] tests/ui/layout/enum.rs stdout ----
diff of stderr:

-	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIGN }
+	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN }
2	  --> $DIR/enum.rs:9:1
3	   |
4	LL | enum UninhabitedVariantAlign {

Note: some mismatched output was normalized before being compared
-	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: Align(8 bytes) }
-	  --> /home/jyn/src/rust2/tests/ui/layout/enum.rs:9:1
+	error: align: AbiAndPrefAlign { abi: Align(2 bytes), pref: $PREF_ALIN }

compiletest now allows using a specific debugger when running debuginfo tests

For a long time, the tests/debuginfo test suite could only be successfully run if you had all the tested debuggers (being lldb, gdb, cdb). This is very annoying locally if:

  • One or more of these debuggers are not available or don't work locally2.
  • You just wanted to look at the test failures for a given debugger.

Now, you are able to run the tests/debuginfo test suite with a specific debugger, to only run the tests against that specified debugger. Example usage:

$ ./x test tests/debuginfo -- --debugger gdb

ui tests now support //@ forbid-output

ui tests can now use the //@ forbid-output: REGEX directive to check for a pattern which must not appear in the stderr. If the REGEX pattern is matched, then the ui test will fail.

Please consult rustc-dev-guide for more details.

./x test now accepts a --no-capture flag which will be forwarded to compiletest (and thus libtest)

Previously, if you wanted to pass the --nocapture flag through bootstrap, through compiletest, to the underlying libtest runner, on Linux you have to write:

$ ./x test tests/ui -- --nocapture

On native Windows msvc it's even worse, I recall having to write

PS> ./x test tests/ui -- -- --nocapture

Which is hard to discover and a contributor papercut3.

Now, you can just write

$ ./x test tests/ui --no-capture

and bootstrap will forward this flag as --nocapture to the underlying libtest.

Notable changes

This section is intended to be like a "compatibility notes" but for human test writers.

FileCheck-based test suites no longer predefine MSVC and NONMSVC FileCheck prefixes

In the current setup, compiletest will register a FileCheck custom prefix for each compiletest //@ revision. However, for historical reasons compiletest also predefined MSVC and NONMSVC FileCheck prefixes depending on the host. But this is very surprising for two reasons:

  1. It's "spooky action in a distance" because the user never declared these custom prefixes, and these prefixes are conditionally set based on the host. It's hard to debug too.
  2. If the user also wants to add their own //@ revision: MSVC NONMSVC revisions, because compiletest would pass --check-prefix for those names twice, this will cause FileCheck to report an error about duplicate prefixes.

Therefore, in compiletest: don't register predefined MSVC/NONMSVC FileCheck prefixes #134463 we no longer predefine these two FileCheck prefixes.

If you want the previous MSVC vs NONMSVC behavior, you will need to explicitly write out

//@ revisions: MSVC NONMSVC
//@[MSVC] only-windows-msvc
//@[NONMSVC] ignore-windows-msvc

normalize-{stderr,stdout}-test directives are renamed to normalize-{stderr,stdout}

Mostly a cleanup, the -test suffixes provide no additionally useful information, and only make these two normalize-* directives hard to discover.

normalize-{stderr,stdout}-test directives are now renamed to normalize-{stderr,stdout}. normalize-{stderr,stdout}-{32,64}bit directives remain unaffected.

compiletest will now deny usages of builtin FileCheck suffixes as revision names (for FileCheck-based test suites)

For FileCheck-based test suites (assembly, codegen, mir-opt), compiletest will register a custom FileCheck prefix for each compiletest //@ revision. However, FileCheck also has some builtin suffixes such as:

// COM:
// CHECK:
// CHECK-NEXT:
// CHECK-SAME:
// CHECK-EMPTY:
// CHECK-NOT:
// CHECK-COUNT:
// CHECK-DAG:
// CHECK-LABEL:

When combined, this previously meant that the compiletest revision + builtin FileCheck suffix constructions such as

// next:
// same-SAME:
// not-NOT:
// next-NEXT:
// not-SAME:

are permitted by compiletest, which are incredibly confusing.

As such, compiletest will now reject CHECK, COM, NEXT, SAME, EMPTY, NOT, COUNT, DAG, LABEL as revision names in FileCheck-based test suites.

PR listing

Improvements

Fixes

Cleanups

Documentation updates

  1. The test infra here refers to the test harness compiletest and supporting components in our build system bootstrap. This test infra is used mainly by rustc and rustdoc. Other tools like cargo, miri or rustfmt maintain their own test infra. ↩

  2. For example, I keep having to debug the debugger like lldb when the older lldb versions keep complaining about ModuleNotFoundError: No module named '_lldb'. ↩

  3. I don't know about you, but I can never for the life of me remember what the flag is called. I keep thinking it's --no-capture when currently libtest only accepts --nocapture. Thankfully T-testing-devex FCP'd to add the more conventional version --no-capture, looking forward to that. I can also never figure out how many -- dashes I need, I keep finding myself having to add more -- until it starts working. ↩

  4. Nyehehehe ↩