This Development-cycle in Cargo: 1.76

Jan. 3, 2024 · Ed Page on behalf of The Cargo Team

We wanted to share what has been happening for the last 6 weeks to better keep the community informed and involved. For work that was merged before branching for 1.76 beta, it will be in the Beta channel for the next 6 weeks after which it will be generally available.

This is distinct from This Week in Rust in that it tries to focus more on the big picture, rather than individual PRs, and pulls from more sources, like Cargo Team meetings and Zulip.

This is an experiment in finding better ways to be engaged with the community and we'll see how well it works and how well we can keep up on it.

Merged

Managing growth of Cargo

The Cargo team has been working to scale our processes to allow the number of packages in the workspace to grow

We've had a couple of breakages affecting people over the last year while we reflected on some recent regressions. Examples include:

  • When making future-incompatible updates to the output of cargo metadata, not coordinating with the third-party cargo_metadata API (oli-obk/cargo_metadata#240)
  • Confusion over dependencies on cargo-credential causing non-working dependency trees when building these packages from the crates.io (rust-lang/cargo#13004)

Some potential improvements include

As a first step in stretching our ability to scale, we split out Cargo.toml serde types from cargo itself (rust-lang/cargo#12801).

Other areas for potentially splitting out of the cargo library include:

  • Move serde and CLI types into cargo-util-schemas
  • Console output
  • Parsing and layer merging for .cargo/config.toml
  • Reading from different package sources (git, path, git registry, sparse registry)
Long-path support

A user ran into path-length issues on Windows with cargo install --git (rust-lang/cargo#13020) which led to ChrisDenton to post a PR for embedding a Windows manifest into the cargo binary, modeled after rustc. After some exploration on that PR, it was merged with rust-lang/cargo#13141 being created to track some of the remaining work (which is in tandem with rust-lang/cargo#9770).

When interacting with git, there are some notes in rust-lang/cargo#13020 on some additional config settings to workaround problems.

Stabilizing cargo metadata's id field

Currently, cargo metadata's package id field is defined to be opaque. The problem is you can't take a package from the output and pass it to cargo <cmd> --package <value>. You could use the name field but that can be ambiguous when there are multiple, incompatible versions in your Cargo.lock. name is a subset of the Package ID Specs format which most --package parameters accept. rust-lang/cargo#12914 proposes we switch id to be Package ID Spec and declare it as non-opaque in cargo metadata's output, allowing a caller to take the id and pass it cargo via the --package parameters.

We did find one hurdle: Package ID Specs can sometimes be ambiguous. That was resolved in rust-lang/cargo#12933.

This is waiting on input from the Cargo team.

-Ztrim-paths

-Ztrim-paths is an unstable feature that provides different options to sanitize paths embedded in final binaries. This improves privacy and reproducibility when shipping and sharing artifacts without sacrificing the debugging experience.

-Ztrim-paths is generally usable and weihanglo has been driving the effort to get this stable. Recently, they fixed an issue when sanitizing the package in the current working directory (rust-lang/cargo#13114) and added end-to-end tests to ensure the debugging experience does not regress (rust-lang/cargo#13091 and rust-lang/cargo#13118).

There are some symbols that are not sanitized yet, for example N_SO and N_OSO symbols on macOS or DW_AT_comp_dir on Linux when split-debuginfo enabled. This is tracked in rust-lang/rust#117652.

When sanitizing paths, we remap the start of the path to an identifier. The current remap rules make it difficult to configure a debugger to remap to the source on their system. Alternative remap rules are being discussed in rust-lang/cargo#13171. An important consideration being raised is that the users can successfully remap in their debugger regardless of endianness or bit-width which is important for cross-platform debugging.

-Zcheck-cfg

-Zcheck-cfg is an unstable feature that will cause rustc to warn on undefined conditional compilation, like #[cfg(unknown)] or #[cfg(feature = "unknown")].

Urgau has been working across rustc and cargo to polish up this feature for stabilization. Recently, they:

Urgau is hoping to start stabilization discussion during the 1.77 development-cycle.

RFC #3516 (public/private dependencies)

RFC #3516 (public/private dependencies) was merged which will help users identify when they leak their dependencies in their public API, helping prevent unintentional breaking changes. This is behind cargo-features = ["public-dependency"]. A good amount of the implementation was done as part of the superseded RFC #1977.

linyihai has stepped in to help implement the remaining Cargo work, including:

Other Cargo work includes

The hope is to have this ready for 2024 Edition. The tracking issue enumerates what work is remaining. The biggest risks are likely:

User-controlled cargo diagnostics

The Cargo Team is very cautious about adding warnings to Cargo because there is nothing like rustcs --allow ... / #[allow(...)] to suppress them when needed. This changed with the introduction of the [lints] table. We are tracking cargo warning control (and the lints it can unblock) in rust-lang/cargo#12235.

The first milestone is for TOML parse errors to match rustc's error style, going from

error: failed to parse manifest at `[..]`

Caused by:
  TOML parse error at line 6, column 25
    |
  6 | build = 3
    | ^
  invalid type: integer `3`, expected a boolean or string

to

error: invalid type: integer `3`, expected a boolean or string
--> Cargo.toml:6:25
  |
6 | build = 3
  | ^

Rather than writing our own error message renderer that imitates rustc, Muscraft resurrected the annotate-snippets project with the intention of making it work for cargo and then migrating rustc to it. They released annotate-snippets v0.10 and created rust-lang/cargo#13172 for integrating it into cargo when parsing Cargo.toml files.

We will also need to decide what to do about the differences in colors between rustc and cargo. Muscraft has been looking into why rustc's colors were chosen and are preparing a proposal for what both programs should use.

cargo info

We've had a request for a cargo info command for close to a decade. hi-rustin, a regular Cargo contributor, has taken on designing such a command.

You can try it out by running

$ cargo install cargo-information
$ cargo info clap

Ideas and feedback are welcome! See the Issue tracker.

Postponing RFCs

The Cargo team is looking to clean up the backlog of open RFCs.

RFC #3379 (os_version predicates for #[cfg]): I'll defer to the summary on the RFC:

I'm going to propose to postpone this RFC. I think we all agree that this would be a great thing to have, but I think there are some big questions, particularly around how version support of pre-built std works, how it might tie into supporting target requirements, how the version information is determined, etc. Primarily, there isn't anyone on the team who has the capacity at this time to champion this feature.

RFC #3177 ([patch] dependencies using unidiff patchfiles): We think this would be very useful but there are a lot of details to work out and no one on the team is able to help shepherd this effort. Taking a lesson from other teams and from the cargo script eRFC, we felt the best way for this to move forward is for someone to sketch out a rough proposal and then implement it as an unstable feature as an experiment. This experiment would be focused on learning the things we need to figure out what should be in the RFC.

RFC #3287 (native code coverage support): We are very much interested in improving the developer experience around coverage but unclear if that RFC is the right approach (e.g. rust-lang/cargo#13040 includes an alternative). Like with RFC #3177, we need to run an experiment to flesh out the design for this.

RFC #3263 (don't treat pre-releases as compatible with each other): We fully recognize that this is a problem. For example, Clap ran into this with the 3.0 pre-releases and is the reason Clap stopped using pre-releases. However, there were a lot of questions that have remained unresolved within the RFC for the last year and no one on the Cargo team has the availability to help drive these conversations. A viable short-term solution would be to use the proposed cargo linting system to warn users that don't pin their pre-release version requirements with = in their Cargo.toml file. As an alternative for short-term testing of pre-releases, users can [patch] in the dependency's git repo. For more extensive use of immature APIs or behavior, Clap has been exposing them through the convention of unstable- prefixed features that are documented as having no semver guarantees. rust-lang/cargo#10881 proposes native support for unstable features. We recognize this does not help when a library is going through a more extensive breaking change and there is still a place for pre-releases.

Action item: We do need to go back and document the experiment process so we can more easily point people to the expectations for running one.

Design discussions

Meta: 2024 Edition

With the window soon closing for the 2024 Edition, we explored whether there is anything else we should attempt to slip in.

Currently, we have planned:

with the possibility of RFC #3537 (MSRV-aware resolver) being tied to an Edition.

We brainstormed other ideas including:

Disabling of default features: We see this as a part of the bigger picture for making it easier to evolve features, particularly taking built-in functionality and putting it behind a feature without it being a breaking change. This likely doesn't need an Edition on its own but we also talked about if we want to remove default-features = false that that would require an Edition. However, we aren't sure if that is what we want, we would not want to rush that design, and we should have a large deprecation window before the switch.

Cross-compile Doctests: Currently, we skip doctests when using --target and this feature changes it so we start running them. Switching to this behavior is likely to break people. Testing out -Zdoctest-xcompile on rust-lang/rust saw no errors with --target=armhf-gnu and --target=arm-android had a few in std. From a user surprise perspective, we feel like people would be more surprised to find out we are silently skipping doctests rather than being surprised to see compilation errors. Maybe having doctests run (and fail) on an Edition change would be viewed as a bug fix. If we move forward with this, the decision is likely not ours alone as we'd need to stabilize flags in other tools as well. The conversation is ongoing on zulip.

Have profile.*.debug=0 imply profile.*.strip = "debuginfo": When a user disables debug info with debug=0, they will still have debug symbols from std as it is pre-built. By implicitly setting strip = "debuginfo" when debug=0, we'd be closer to what the user actually requested. According to the zulip thread, this speeds up builds on Linux and shrinks binaries. The downsides when debug=0 is set are slightly slower builds on Mac and backtraces will be less informative. To put the backtraces in perspective, this will make backtraces through std consistent with user code and user code is likely to be the majority of an application. We felt this could move forward as-is without an Edition and asked for a more formal proposal which can be found on the issue.

Make profile.*.split-debuginfo the default: We previously changed the default for Mac (rust-lang/cargo#9298) to improve compile times and reduce target-dir size. If we make this change, we realized it can't be tied to an Edition because [profile] is a workspace-level field and Cargo only has the concept of an Edition for packages and not workspace. For further discussion, see the zulip thread.

[lints] redux

1.74 saw the introduction of [lints] in Cargo.toml.

The primary points of concern as more people tried this out were

For the last point, we discussed implicit workspace inheritance for all fields. The main concern we focused on was about the challenges of this being too magical, making it harder for people to reason about what cargo is doing. One piece of prior art is [profile]. While Cargo implicitly layers Cargo.toml over .cargo/config.toml, this is involving configuration which is already a little off the beaten path and might not be the best model to follow. However, it also supports profiles explicitly saying they inherit from another profile. We could have a manifest-wide opt-in for inheriting fields that were not explicitly set. Based on feedback, we could then explore changing the default for this on an edition. One awkward case is dependencies. We shouldn't automatically add all dependencies. We also probably shouldn't allow dependencies without a source (i.e. allow skipping workspace = true). But not having dependencies participate would be inconsistent.

Discussing the prior art of [profile] also led to a discussion of having more than one set of values you can inherit from. We discussed a couple of ideas, including

  • Having named sets of fields that you inherit all-or-nothing (inherits = "public-members")
  • Having named fields that you can inherit per-field (rust-version.workspace = "public-members")
  • Naming other packages you can inherit from, either whole or per field
cargo script

Progress on the cargo script RFC and implementation has stalled while the RFC for embedding manifests is figured out.

The current proposal is:

#!/usr/bin/env cargo
```cargo
[dependencies]
clap = { version = "4.2", features = ["derive"] }
```

use clap::Parser;

#[derive(Parser, Debug)]
#[clap(version)]
struct Args {
    #[clap(short, long, help = "Path to config")]
    config: Option<std::path::PathBuf>,
}

fn main() {
    let args = Args::parse();
    println!("{:?}", args);
}

The manifest is embedded in what we are calling a code-fence frontmatter, modeled off of markdown. The cargo identifier is referred to as an infostring.

There are two directions we can take the infostring in the long run:

  • Does the parent tool (in this case, cargo) own the definition of the infostring and is allowed whatever identifiers it wants
  • Does rustc own the meaning of the infostring, allowing the Rust Project to add additional types of metadata without concern for breaking tools that rely on custom identifiers

The embedded manifest syntax RFC was updated with a new section, side-stepping this discussion by suggesting we hard code support for cargo right now and leave the decision to the future when we have more context for how this might be used.

We also recognize that using three backticks would likely trip users up when they try to put these into markdown code fences as users might not be aware of how to escape it or forget to escape it, causing frustration.

There is on-going discussion on zulip.

SBOM

Supply-chain security is getting a lot of attention lately. One element of this is being able to trace what all was pulled into a binary and how it was built. This is referred to as a Software Bill of Materials, or SBOM.

Previously, arlosi created a Pre-RFC on this topic. The Pre-RFC has continued to garner discussion, including

  • Enumerating the limitations of third-party solutions (link)
  • On the role of Cargo's SBOM format, whether it should be an intermediate format that gives integrators the information they need to create their final SBOM or whether it should be self-contained enough to just do a format conversion to the format of choice. This affects fields like
    • Author (can be pulled from manifests)
    • Hashes (which comes with questions of which artifacts using which algorithms)
    • Timestamps (which is not reproducible which is a requirement for some SBOM use cases)

arlosi is planning to incorporate the feedback into the Pre-RFC, do a last call for feedback, and move onto a full RFC.

RFC #3537: Make Cargo respect minimum supported Rust version (MSRV) when selecting dependencies

One frustration point with Cargo and the crates.io ecosystem is when a dependency is added and the build fails because it uses newer features than your version of Rust supports. We've been tracking this in rust-lang/cargo#9930. Historically, we've deferred work on this as we expect the errors when no compatible package is present to cause more confusion and user frustration than the problem being solved. We were hopeful that a PubGrub dependency resolver would fix this but there is a lot of work remaining before we can switch dependency resolvers.

During the 1.74 development-cycle, we merged rust-lang/cargo#12560 which added a perma-unstable implementation so people could at least use nightly for one-off dependency resolution. Just before the 1.76 development-cycle, we came up with a way to side-step the resolver error messages by only preferring stable versions which we merged in rust-lang/cargo#12950.

This side-step was written up as Pre-RFC: MSRV Aware Resolver which led into RFC #3537: Make Cargo respect minimum supported Rust version (MSRV) when selecting dependencies.

One of the main conversations is about whether we should respect MSRV by default or require an opt-in. After some discussion during cargo team meetings and office hours, the plan moving forward is to re-focus the document on workflows, what behavior we want to drive (e.g. avoiding stagnation), and how different possible solutions affect the workflows and user behavior.

We are waiting on the RFC author for integrating this new approach into the RFC.

RFC #3371: CARGO_TARGET_BASE_DIR support

RFC #3371 allows users to move all of their target-dirs to be under a common root directory without a lot of bookkeeping on the users end. It was proposed for merge back in June but it fell off our radar and we were finally able to talk over it. We clarified that this proposal is independent of per-user caching and both efforts are mostly independent and worthwhile. Per-user caching would reduce how much we put in target-dir but workspaces would still need a target-dir for uncacheable builds and final artifacts.

While there are other solutions that cover the motivations for CARGO_TARGET_BASE_DIR, we felt CARGO_TARGET_BASE_DIR is a principled shortcut that can get us most of those benefits sooner.

One concern raised in the RFC is how can people find their target-dir (e.g. packaging their built [[bin]]s). This becomes more relevant if we were to consider switching the workspace's target-dir to a central location. In the RFC, the idea of symlinking target/ to target-dir was brought up. It is unclear whether the benefits to those that need it outweigh the annoyance for those that don't. Users can get the location of target-dir via cargo metadata. Stabilizing --out-dir would bypass most uses for accessing target-dir. Those might be sufficient. If not, maybe we could explore having a config field to control the creation of a symlink.

We then explored the design space a little bit, taking inspiration from the index's dl field. For example, having placeholders for {home} or {cargo_home} would make it easier to copy configs from account to account. What if we extend CARGO_TARGET_DIR with placeholder support instead of adding CARGO_TARGET_BASE_DIR, allowing CARGO_TARGET_DIR={cargo_home}/target/{manifest_path_hash}? This seems like this would simplify the design quite a bit.

This is now back on the RFC author's plate to process this feedback and update the proposal as they see fit.

RFC #3493: cargo update --precise <prerelease>

To use a pre-release today, users have to opt-in with their version requirements. RFC #3493 changes cargo's dependency resolver so that users can opt-in to a pre-release in their Cargo.lock. This works well if users want to test for regressions in a pre-release dependency or want access to functionality early but don't require it (e.g. performance improvements). If a package requires something from a pre-release, like a new API, that should instead be specified in the version requirement which is more of the focus of RFC #3263.

Since we were already discussing postponing #3263, we discussed whether we should also postpone #3493. While we want to improve pre-releases, no one on the team is available to help shepherd this and the proposal would involve an invasive change to cargo that would likely be brittle. For how much time we do take to address pre-release, it was unclear if this was the most important.

As we discussed it, we realized there was a solid precedence to base the design off of, yanked packages. We could also minimize risk by suggesting that the semver package keep the existing version matching logic and expose the new behavior under a different function name. We also realized that this RFC is a prerequisite for RFC #3263 so that cargo could correctly unify pre-release and regular release version requirements.

For bookkeeping purposes, there was a concern this would be require Cargo.lock changes. If so, then it would likely be complex enough that we'd need an experimental implementation first. However, we found Cargo.lock changes are unlikely to be needed after some further discussion.

We updated the RFC and this is now waiting on author to wrap up the discussion.

cargo update --precise <yanked>

Between RFC #3493 letting users force pre-releases through --precise and rust-lang/cargo#12425 doing the same for breaking changes, it seemed fitting to extend this concept to yanked packages, resolving rust-lang/cargo#4225. We felt we need to trust users in these scenarios. Users might have valid reasons to access yanked packages, whether its short-term to test something out or long term and they accept the risks. We considered an additional flag for this but pre-release and breaking changes don't need a flag. For breaking changes, the flag would be for use outside --precise for opting in for all packages. There is the possibility we'd want to extend that concept to pre-release and yanked packages. For a preponderance of caution, we reached out to a prior cargo team member in case there was context we were missing and they gave the greenlight.

rust-lang/cargo#4225 is marked as accepted and we welcome people to contribute a PR for this.

Misc

Other relevant topics of interest:

Focus areas without progress

These are areas of interest for Cargo team members with no reportable progress for this development-cycle.

Ready-to-develop:

Needs design and/or experimentation:

Planning:

How you can help

If you have ideas for improving cargo, we recommend first checking our backlog and then exploring the idea on Internals.

If there is a particular issue that you are wanting resolved that wasn't discussed here, some steps you can take to help move it along include:

  • Summarizing the existing conversation (example: Better support for docker layer caching, Change in Cargo.lock policy, MSRV-aware resolver )
  • Document prior art from other ecosystems so we can build on the work others have done and make something familiar to users, where it makes sense
  • Document related problems and solutions within Cargo so we see if we are solving to the right layer of abstraction
  • Building on those posts, propose a solution that takes into account the above information and cargo's compatibility requirements (example)

We are available to help mentor people for S-accepted issues on zulip and you can talk to us in real-time during Contributor Office Hours. If you are looking to help with one of the bigger projects mentioned here and are just starting out, fixing some issues will help familiarize yourself with the process and expectations, making things go more smoothly. If you'd like to tackle something without a mentor, the expectations will be higher on what you'll need to do on your own.