This Development-cycle in Cargo: 1.80

June 19, 2024 · Ed Page on behalf of The Cargo Team

This Development-cycle in Cargo: 1.80

This is a summary of what has been happening around Cargo development for the last 6 weeks which is approximately the merge window for Rust 1.80.

Plugin of the cycle

Cargo can't be everything to everyone, if for no other reason than the compatibility guarantees it must uphold. Plugins play an important part of the Cargo ecosystem and we want to celebrate them.

Our plugin for this cycle is cargo-expand a more convenient and easier to remember wrapper around cargo rustc for seeing all macros expanded within a crate. For macro authors, this is a big debugging help. For macro users, this can help in breaking the opacity of what a macro invocation does.

Thanks to LukeMathWalker for the suggestion!

Please submit your suggestions for the next post.

Implementation

-Zcheck-cfg

Update from 1.77

As a refresher, this is a rustc feature that checks #[cfg]s against a list of known names and values. When used with Cargo, the names and values come from:

At the beginning of May, Cargo's support for --check-cfg was stabilized (#13571). When stabilizing, the team weighed the crater results and feedback from the multiple rounds of Call for Testing (which included tweaking the layout for This Week in Rust to improve visibility). Hand-in-hand with stabilization, urgau published a blog post to help people first exposed to this by upgrading nightly.

Soon after this hit nightly, rust-lang/rust#124800 (and related issues) were open. People's concerns included:

  • The false positive rate
  • Having to use a build.rs for custom --cfgs where one wasn't used before
  • #[cfg]s in build.rs itself which has no way of specifying --check-cfg for itself
  • Some confusion over the lint message and documentation

A lot of discussion was spent on trying to get everyone involved in the conversation (including us) onto the same page in terms of what this feature was, what the options were for working with it, and the impact of each of those. This is understandably a frustrating process because people who are negatively impacted are feeling the pain now. However, we need to make sure we find the right solutions rather than the first.

A positive of this was that nightly was doing its job in helping to collect critical feedback well before we hit stable! To give us plenty of time for feedback, we intentionally held back the stabilization PR until after the previous nightly was branched to beta, so we'd have a full 12 weeks to collect feedback and improve this. As this was a lint (non-blocking) and we felt confident in doing any needed polish in the 12 weeks before release, we decided to keep this in nightly, rolling back only if we were running up to the deadline for release.

We recognize that the documentation was an issue. We worked together to find ways to improve it (e.g. #13869 #13937 rust-lang/rust#124209 ). One challenge that limited this work was finding a place for Cargo's documentation to live as there isn't a user-focused Cargo feature for this to center the documentation around.

As for improving how people interact with this feature, longer term we feel private features and mutually-exclusive, global features would help replace a lot of cases for custom cfgs. That still leaves the short term.

An straightforward answer that came up multiple times was to add to Cargo.toml a [cfg] table (#11631). This was proposed during the development of the feature but was rejected by the Cargo team as designing a whole new system that didn't align with the rest of cargo (see also Leaky abstractions of rustc).

Through a mixture of Github, Cargo Office Hours, and in-person conversations at a conference, we settled on the solution of using lint configuration from the "Future Possibilities" of RFC 3389: [lints]. Shout out to wesleywiser for the idea and urgau for the implementation in #13913.

e.g.

[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(has_foo)'] }

(see Cargo Specifics - Checking Conditional Configurations for more details)

We weren't necessarily done and had to address questions like:

  • Should we prevent people from extending Cargo check-cfg values?
  • Should we reduce warning noise in 1.79 as part of the transition to this feature? Yes, in #13925.
  • Are we okay with the inconsistent dashes in lints.rust.unexpected_cfgs.check-cfg? Yes, its not too different from dependency names and fields.
  • Should we allow the level to be optional? As its an MSRV bump either way, we decided to defer making a decision on this. Some lean towards preferring being explicit.

One lesson learned from this is that the Cargo and Compiler teams should better coordinate on changes like this. In particular, public-private dependencies seems like it will have a similar effect on the ecosystem.

User-controlled cargo diagnostics

Update from 1.79

This development-cycle, diagnostics mostly saw polish.

We did have a forward-looking conversation among the Cargo team: when should lints be evaluated?

Most manifest errors and warnings today are processed when the document is parsed. Warnings get captured and reported later only in some commands and only if we don't otherwise cap the lints for that package. Or put another way, we analyze warnings for every package in your dependency tree and then throw away almost all of that work.

If we were to move some of these errors and warnings to when we run our new diagnostics, we could stop doing throw-away work and maybe improve some aspects of code organization. As a side effect, some errors that get reported today might not get reported at all. Could we change this later without breaking our expectation for compatibility or, if we can't, are we ok with being stuck with this new behavior? After some discussion, we felt that we could move in this direction but we'll need to look at this on a case by case basis.

-Ztrim-paths

Update from 1.76

rust-lang/rust#107099 was merged, allowing rustdoc to participate in trim-paths. Cargo still needs to be updated.

Another challenge Cargo has had is access to a hash algorithm that is stable across platforms and not just run-to-run. Urgau is working to pull out rustc's stable hasher for Cargo to reuse (rust-lang/rustc-stable-hash#1).

MSRV-aware Cargo

Update from 1.79

The main focus at this point is the MSRV-aware resolver.

We ran a Call for Testing and so far we haven't gotten any actionable feedback. Separate from that Call for Testing, we did receive feedback about the latest annotations in cargo update (#13908). Discussion is still on-going.

As far as we are aware, the last remaining step before stabilization is renaming the configuration (#13540):

[resolver]
something-like-precedence = "something-like-rust-version"

We want to keep in mind how this might align with future possibilities like:

For minimal vs maximal version resolution, we see that is likely an enumerate value under an unnamed key, so we can set that aside.

For the rest, we are talking about the following modes:

mode MSRV yanked prerelease
this is yet another candidate Required Never? Never?
de-priorize this over other versions Required Likely Likely
don't resolve to if already in use Likely Required Required

This helped to show that we probably want to name the field incompatible-rust-version to clarify that we are talking about how we are handling those packages and to leave room for resolver.rust-version to override what version is used when resolving dependencies.

The challenge is providing a clear way to communicate the "mode". For de-prioritizing, we considered:

  • discourage
  • eschew
  • avoid
  • fallback

Of those, fallback most captures the nuance but that is a noun while we had been considering allow and deny for the other modes which are verbs.

We had considered a short-term solution of a "precedence policy" field that would take a policy name, much like we have profile names that encompass many compiler settings. This would be short term and could run into its own issues with being confusing, so we decided to focus on the "right" solution.

Removing implicit features

Update from 1.78 and 1.79

A crater run exposed a couple of bugs with cargo fix when migrating implicit features to explicit features (#14010).

This exposed a misunderstanding that ["dep_name/feature_name"] is not always the same as ["dep:dep_name", "dep_name?/feature_name"] because the former won't suppress the creation of an implicit feature. As a side effect of how we prevent implicit features from being created in Edition 2024, there are corner cases where the ["dep_name/feature_name"] syntax will error (#14016) . We considered whether we should deprecate the syntax (which we had previously deferred) or whether to implicitly inject "dep:dep_name". After some discussion, we settled on the latter.

Normalizing published manifest files

Update from 1.79

Now that published Cargo.toml files list all targets, we can avoid reading the filesystem with #13849.

Thanks for stormshield-guillaumed testing nightlies, we found out that these changes made the vendored Cargo.toml non-deterministic and fixed that in #14004.

Merging cargo upgrade into cargo update

Update from 1.77

In #13979, torhovland added support for cargo update --breaking with the following policy:

  1. In-memory, upgrades certain workspace member dependencies to the latest breaking version
  • Must use ^ version requirement operator (the default)
  • Must not be renamed as those tend to be used to allow multiple versions of the same package and the user likely doesn't want the version changed
  • Can further limit to specific dependencies by naming them
  1. Reconcile the lockfile with these new version requirements
  2. Write out the new version requirements, preserving the precision used in the original requirement (e.g. 1.0 would be upgraded to 2.0, not 2.0.5)

This is available to use as of nightly 2024-06-09.

.crate provenance

With the xz backdoor, there has been an increased interest in verifying that a published .crate matches the repo. Cargo already includes a cargo_vcs_info.json file in the .crate to identify what cargo publish was run against. One problem is that this file wasn't being generated if --allow-dirty was used (#13695).

After some discussion, torhovland created #13960 for us to always generate cargo_vcs_info.json but to include a dirty: "true" field when --allow-dirty was used.

For more investigation on using this file, see 999 crates of Rust on the wall.

cargo publish --workspace

torhovland and jneem stepped up to break down what it would take to add --workspace in #1169 and #10948. They started on the milestone of support for cargo package --workspace.

The first problem to address is how will we generate a valid lockfile for packages that haven't been uploaded. We discussed this in Office Hours and decided to implement an internal-only package-source overlay system in #13926. This would allow local packages to pretend to be on crates.io. This might sound like a very useful future but we intentionally decided to keep this internal-only and only implemented it with great hesitance because a generalized version of this would be a source of dependency-confusion attacks.

Now that the overlay-source is implemented, work continues on #13947.

Snapshot testing

Update from 1.78

Cargo has had a homegrown CLI assertion framework with support for features like

  • Redactions
  • Jsonlines
  • Unordered text and jsonlines
  • Pretty diffs on failure

As there is less of a community around this, the knowledge is more specialized, its less likely to be documented, and we are on our own for feature development.

epage and Muscraft have been generalizing the concepts from cargo-test-support into snapbox. This has been used for UI testing in cargo since cargo add.

Out of frustration with hand-editing nearly every test multiple times this year, epage dove into closing the functionality gaps with cargo-test-support and release this in the 0.5.11 and 0.6.0 releases.

We started the porting effort with non-CLI assertions in #13980 and then CLI assertions in #14031.

Immediate benefits for cargo contributors

  • Update test results by adding env SNAPSHOTS=overwrite
  • Auto-redact non-deterministic snapshotted values (e.g. [ELAPSED]) or platform-specific snapshotted values (e.g. [BROKEN_PIPE])

We've documented the process and opened the porting work for anyone to contribute in #14039, like weihanglo did in #14041.

Design discussions

RFC triage

When discussing the 2024 Edition, it came up that an RFC was partly delayed because the author and the reviewing team each thought they were waiting on the other. RFC authors have a real need driving them to write a proposal and put a lot of time into writing the RFC and driving the discussion. Out of respect for investment that RFC authors have put in, epage wanted to drive another pass in trying to make sure there is a clear owner for the next steps for each RFC. This might even include closing the RFC which can be a sensitive topic.

For our first, epage reminded us that templating CARGO_TARGET_DIR has been proposed to merge and is waiting on team members to review it.

RFC #3383: recommended-bins: This was briefly touched on when we last discussed when to use a package or a workspace. Since then, the #[diagnostic] attribute was stabilized in 1.78. Mirroring the concept in Cargo would be a lot lower of a barrier to entry than a fully designed feature. The Cargo team got behind this idea and shared this on the RFC.

RFC 3310: root rustflags: This was originally written for coverage reporting and later the author wrote #3287. That said, the feature seems generally useful, even if dangerous (see the Drawbacks). In the end, we decided to postpone this as the author hasn't been responsive. We included next steps for someone to pick it up.

RFC 3416: Features as table, not just array: Overall, we were in favor of this moving forward. The main concerns were naming (which we worked out) and ensuring third-parties have a heads up of the potential change to Cargo.toml (announced on Zulip). As a reminder, we provide the definition of TomlManifest for others to pull in and use.

Custom test harnesses and panic = "abort"

#11214 is looking for a way to allow panic = "abort" in tests and benches. For example, ideally benches reflect what the production code will do but if the production code is built with panic = "abort" but the tests are built with panic = "unwind", then they can have different performance characteristics.

panic is a profile setting and gets applied to all build targets. We don't want someone changing this value for their bins to affect their tests unexpectedly.

We also need to keep in mind backwards compatibility. Changing this setting would immediately break people.

We didn't come to any immediate conclusion on how to move forward with this.

Short-hand manifest syntaxes

RFC #3502 was recently merged. Throughout its life, we've occasionally brainstormed ways to smooth out the feature more, like supporting package.edition as a number and not just a string.

Removing the need for [package] header: Instead each package field would work in the top-level of the manifest. For cargo-script, this would mean you could do

edition = "2024"

instead of

package.edition = "2024"

or

[package]
edition = "2024"

The main concerns raised were

  • Detecting if the package is present when using [workspace]
  • Potential implementation complexities, especially around good error reporting
  • If we simplified this by limiting it to cargo-script, it would add more steps to by-hand convert a cargo-script to a multi-file package

Overall, this is something that doesn't have to be decided now. Instead, we can see how people use cargo-script and decide what improvements are needed.

Embedded build.rs: The following Cargo.toml alternatives to a standalone build.rs were proposed:

build.rs = '''
fn main() {
  ...
}
'''
# or
build.rs.output = '''
cargo::directive=...
'''

While build.rs.output might have helped with check-cfg before we added lint configuration, it seems pretty rare that it would be of use.

Embedding rust source in the manifest would allow a cargo-script to have a build.rs. While having a fully separate -sys package is reasonable for general cases, there can be one-off "I just want a lib for a little bit" use cases. We had previously decided that embedding other content in a cargo-script (build script, config, proc-macros, addition source and packages), was an anti-feature and we'd need to revisit a lot more than just this. Instead, we would want to encourage multi-file packages in these cases.

As an alternative, we discussed metabuild which has been stuck in unstable limbo due to the lack of real world use and polyfills like system-deps.

Again, this is likely something we should wait until cargo-script has been stabilized and in use for a while.

Leaky abstractions of rustc

Cargo is dependent on the behavior of rustc but frequently users regularly need access to rustc features that haven't been abstracted yet (#12739), leading to RUSTFLAGS and cargo rustc.

Here be dragons though. A recent example of this causing user confusion is that RUSTFLAGS=-Copt-level=3 cargo test will disable debug assertions while profile.test.opt-level = 3 does not (#14033).

Similarly, cargo build --message-format=json does not report back all json messages from the compiler. This makes life more difficult for cargo show-asm as they want to tell the compiler to emit the assembly and having the compiler tell them where to find the assembly files, rather than trying to guess (#13672). Cargo filters out messages related to implementation details (e.g. emitted files and unstable aspects of target-dir) but doesn't know when a message came back because its tied to an implementation detail or a user requested it via RUSTFLAGS. While people can find ways to rely on those same implementation details, there is a difference between the user knowingly going off the beaten path and us endorsing it in terms of the burden it places on us for evolving things in the future. During the team meeting, we couldn't come up with a solution that satisfied us and we reported back our unsatisfactory ideas on the issue. However, in Office Hour, one idea for a rustc feature for this would be to permit specifying a root directory, and not just a file, to emit to.

Misc

  • zstd .crate files were briefly discussed on zulip, see also #2526.
  • Daily reports by Eh2406 on the progress of the Rust implementation of the PubGrub version solving algorithm
  • More progress has been made on #13709 for RFC 3553: SBOM
  • We've identified more corner cases for cargo update --precise <prerelease> (#13290)

Focus areas without progress

These are areas of interest for Cargo team members with no reportable progress for this development-cycle.

Ready-to-develop:

Needs design and/or experimentation:

Planning:

How you can help

If you have ideas for improving cargo, we recommend first checking our backlog and then exploring the idea on Internals.

If there is a particular issue that you are wanting resolved that wasn't discussed here, some steps you can take to help move it along include:

  • Summarizing the existing conversation (example: Better support for docker layer caching, Change in Cargo.lock policy, MSRV-aware resolver )
  • Document prior art from other ecosystems so we can build on the work others have done and make something familiar to users, where it makes sense
  • Document related problems and solutions within Cargo so we see if we are solving to the right layer of abstraction
  • Building on those posts, propose a solution that takes into account the above information and cargo's compatibility requirements (example)

We are available to help mentor people for S-accepted issues on zulip and you can talk to us in real-time during Contributor Office Hours. If you are looking to help with one of the bigger projects mentioned here and are just starting out, fixing some issues will help familiarize yourself with the process and expectations, making things go more smoothly. If you'd like to tackle something without a mentor, the expectations will be higher on what you'll need to do on your own.