This Development-cycle in Cargo: 1.85

Jan. 17, 2025 · Ed Page on behalf of The Cargo Team

This Development-cycle in Cargo: 1.85

This is a summary of what has been happening around Cargo development for the last 6 weeks which is approximately the merge window for Rust 1.85.

Plugin of the cycle

Cargo can't be everything to everyone, if for no other reason than the compatibility guarantees it must uphold. Plugins play an important part of the Cargo ecosystem and we want to celebrate them.

Our choice for this cycle is clippy-sarif. While not a plugin, it wraps cargo clippy, converting the lint output into Static Analysis Results Interchange Format (SARIF) which GitHub can use to post comments on a PR review for lints, making them easier to access than reading a log file.

Thanks to epage for the suggestion!

Please submit your suggestions for the next post.

Implementation

Rustflags and caching

Update from 1.84

As a recap, Cargo throws out the cache on changes to RUSTFLAGS, instead of creating separate entries in the cache. The problem is RUSTFLAGS is opaque to Cargo and different flags have different caching needs. The remaining problem is --remap-path-prefix which is used to make two different builds look the same. The parameters to the flag are unique to each build and the cache key ends up in the final binary, preventing --remap-path-prefix from serving its purpose.

We talked about this a Cargo team meeting. Once we have built-in support for --remap-path-prefix through trim-paths, we could just point people to that instead. However, there are open issues with trim-paths and its unclear how long that will take to become available. We also could go back to the fix for this from the previous attempt and parse RUSTFLAGS (see #6966) but we're uncomfortable with that because parsing of command-lines is context-sensitive. We ended up with a middle-ground where we apply a rough heuristic where if something looks like --remap-path-prefix, we do not include RUSTFLAGS in our cache key.

This was merged in #14830

Report progress in the taskbar

Previously, Gordon01 picked up work on ConEmu's OSC 9;4 which is an ANSI escape code they defined to report progress through the Windows taskbar icon (#14615). Support for reporting this has extended to applications like systemd (systemd#34929) while support for reporting this has extended to terminals like Ptyxis.

With an FCP started, the team briefly discussed whether this needs to be added to .cargo/config.toml. We previously added configuration control for other terminal features, like unicode and hyperlinks. We don't exclusively turn on these features because not all terminals gracefully degrade in their presence. In particular for the Taskbar escape code, Kitty had made their own feature using OSC 9, causing a poor experience with OSC 9;4 (kitty#8011). This gives us a way for users to test the feature in case their terminal isn't in our allow list or to turn it off if they don't like the results.

The discussion then moved to #14615, particularly focusing on what to name the config field. Currently, the field is proposed to go in the term.progress table. As these are de facto standards, there is no officially defined name for this feature. We could name it for the escape code (term.progress.osc9_4), the original intent (term.progress.taskbar), how progress is being reported (e.g. term.progress.ansi), or one of a myriad of options.

Ideally, the config field would be:

  • Easy discover for people wanting to change this behavior
  • Clear meaning when looking at a config or config docs
  • Consistent with the wider community

At this time, we have not come to a consensus on a name.

cargo publish dirty-check performance

When running cargo package or cargo publish, Cargo records the git commit hash and path within the repo into a file in the .crate file. This was skipped if --allow-dirty was used which changed in Cargo 1.81 to help with efforts in auditing published packages (#13960).

landonxjames reported that this change caused a significant perfornce regression when publishing the aws-sdk-rust repo. Cargo went from skipping its dirty file check with --allow-dirty to always running it. The check has not been optimized for publishing over 400 packages at once. For example, Cargo checks if any file in the repo is dirty. weihanglo reduced the scope of this check to just the package directory in #14962 but we held off on that change as this drew attention to shortcomings in the dirty check (#14967) and we were concerned that fixing those problems would conflict with the performance improvement.

weihanglo set out to test that theory by fixing several of the problems in #14962, particularly for files that Cargo copies into the package root ( #14981, #14966 ). They tried to further optimize this in #14985 though it didn't seem worth the complexity.

With the approach taken above, path filtering isn't an issue so we moved forward with it in #14997.

We are waiting to hear back to see if more work is needed for optimizing this.

Future-proofing the Index

When discussing new features for Cargo, changes to the Index come up. The Index is a collection of basic metadata (called Summaries) for every package version in a registry to allow dependency resolution without downloading a .crate file and parsing the Cargo.toml. Purely additive changes to a Summary are fine, like adding package.rust-version. We can extend existing fields or change the interpretation of them but the experience isn't ideal for old versions of Cargo.

Back when Cargo 1.60 was released, users on older versions of Cargo started to see errors like

error: failed to select a version for the requirement `libgit2-sys = "^0.13.3"`
candidate versions found which didn't match: 0.13.2+1.4.2, 0.13.1+1.4.2, 0.13.0+1.4.1, ...
location searched: crates.io index
required by package `git2 v0.14.3`
    ... which satisfies dependency `git2 = "^0.14"` (locked to 0.14.3) of package `repor v0.1.0 (/home/epage/src/personal/repro)`

When you go on crates.io, version 0.13.3 exists. The problem is 0.13.3 used the new weak dependency feature syntax (dep?/feature) which older Cargos do not understand. This error message is a poor user experience for weak dependency features and any future extensions of the Summary schema. We've been tracking this usability problem in #10623.

epage did some recent work ( #14897, #14921, #14923, #14927 ) so Cargo can extract as much information as it can from unsupported Summaries, without losing this information through caching, and report it to the user.

The following output shows three different ways we might report an unsupported Summary, depending on how much context we have:

[UPDATING] `dummy-registry` index
[ERROR] failed to select a version for the requirement `foo = "^0.1.1"`
  version 0.1.3 requires cargo 1.2345
  version 0.1.4 requires a Cargo version that supports index version 1000000000
  version 0.1.5's index entry is invalid
location searched: `dummy-registry` index (which is replacing registry `crates-io`)
required by package `a v0.5.0 ([ROOT]/foo)`

We also report yanked versions now:

[UPDATING] `dummy-registry` index
[ERROR] failed to select a version for the requirement `baz = "=0.0.2"`
  version 0.0.2 is yanked
location searched: `dummy-registry` index (which is replacing registry `crates-io`)
required by package `bar v0.0.1`
    ... which satisfies dependency `bar = "*"` of package `foo v0.0.1 ([ROOT]/foo)`

With the improved error reporting, we are more free to design Index related features.

Design discussions

Project goals

Cargo-related goals that have been proposed for 2025h1 include

Automatic retry for cargo publish

ijackson proposed that cargo automatically backoff and retry when hitting crates.io rate limits in #13530.

The two main concerns we had were:

Does this follow the intent of crates.io in setting a rate limit? We reached out and they said they are fine with it from a technical perspective.

Is this the user experience we want? The retry times are quite long and users might be unhappy with cargo publish unexpectedly "hanging" for several minutes, even worse if they are publishing a workspace (#1169). If the user decided to cancel a workspace publish with ctrl-c, they could be left in a half-published state. Having an --idempotent flag to skip versions already published (#13397) could help. If this is part of a larger release process, like with cargo-release, then recovery on failure can be messy. In cargo-release's case, it guesses whether an operation will be rate limited and blocks the user, requiring them to override it. We did not look into what canceling a CI-based release, like with release-plz, would look like. Instead of cargo-release guessing whether the rate limit would be hit, crates.io could provide an API to query this however the crates.io team is worried about the compatibility burden this may put on them to further evolve the rate limits.

When crates.io came back with their feedback on this, they did bring up the idea of adjusting the crate-version rate limit from being scoped per-user to being per-crate. This is being tracked in crates.io#10396.

cargo publish verify behavior

When testing cargo publish --workspace (#1169), epage realized that the verify step is running cargo build, rather than cargo check. While this likely doesn't matter for publishing, it can have a dramatic effect when iterating on a release process with the --dry-run flag. They proposed this get changed in #14930 but after some back and forth, it turned out there might be more nuance in this and the discussion shifted to #14941.

Something to keep in mind is that cargo check is not guaranteed to find every compiler error that cargo build may find (see RFC #3477). More specifically, these cover post-monomorphizaton errors which includes linker errors.

We discussed this in the recent Cargo team meeting. Already someone has asked for verification support to be configurable so it can run tests (#14685). If cargo had that, this only becomes a question of defaults. Without it, we would be locking people into an answer which puts more pressure on the decision. However, there are several questions to work out for #14685 that could slow down this issue if we block on it.

An important question for us to answer is what the default purpose of the verify step should be

  • Sanity check packaging (e.g. were all needed files included)
    • If you consider testing of a .crate file important, then running tests is the only way to sanity check it because they can rely on data files, see #14685 for more details
    • For -sys crates, there is the possibility of a missing file causing linker errors
    • Only checking one combination of features or one profile could also mask problems with missing files
  • Last-ditch baseline quality check
    • In case no CI
    • In case changes were made locally before publishing, particularly with --allow-dirty

If its a last-ditch baseline quality check, then running build adds extra assurances in light of RFC #3477. The question then becomes, what is the right default baseline?

  • A fast "most build problems" check
  • Current: The default feature on a dev profile on the current platform
  • Different feature combinations?
  • Different platform combinations?
  • Different profile combinations?
  • Linting, testing, unsafe checking, or formal proofs

Whether the differences between cargo check and cargo build are relevant depends on factors like:

  • Our general assumption of what users do before publishing (cargo test? run all of CI?)
  • The likelihood that a step from above is missed before release
  • The likelihood of post-monomorphization errors being introduced in that gap
  • The number of affected users if a problem does slip through (are more popular crates likely to have more rigorous processes?)

The scariest part of rust-lang/rfcs#3477 is that a crate that builds with cargo check is not subject to Rust's stability guarantees, only cargo build. However, a crate that can only be checked and not built is of little use and is unlikely people could depend on that.

We decided to reach out on Internals to get more feedback on the role of verify and the impact of its performance.

Codifying build script patterns

Inspired by our interest in finding abstractions for RUSTFLAGS (#12739), epage started tracking uses of build.rs in #14948 so we can explore what are gaps in Rust and Cargo that need filling

If nothing else, we can use metabuild for custom declarative build scripts. To simplify the design and to align with other features, epage posted a counter proposal that builds on artifact dependencies. Compared to the current design, this avoids duplicate build and linking of build script binaries and allows finer grained tracking of when to re-run them. This might even allow the running of build scripts to be conditioned on features. joshtriplett suggested consolidating the ways to pass data to build scripts, which epage agreed with. Whats left unclear is whether the data should be low level (literal environment variables) or abstracted (some configuration that gets converted to environment variables).

Misc

Focus areas without progress

These are areas of interest for Cargo team members with no reportable progress for this development-cycle.

Ready-to-develop:

Needs design and/or experimentation:

Planning:

How you can help

If you have ideas for improving cargo, we recommend first checking our backlog and then exploring the idea on Internals.

If there is a particular issue that you are wanting resolved that wasn't discussed here, some steps you can take to help move it along include:

  • Summarizing the existing conversation (example: Better support for docker layer caching, Change in Cargo.lock policy, MSRV-aware resolver )
  • Document prior art from other ecosystems so we can build on the work others have done and make something familiar to users, where it makes sense
  • Document related problems and solutions within Cargo so we see if we are solving to the right layer of abstraction
  • Building on those posts, propose a solution that takes into account the above information and cargo's compatibility requirements (example)

We are available to help mentor people for S-accepted issues on zulip and you can talk to us in real-time during Contributor Office Hours. If you are looking to help with one of the bigger projects mentioned here and are just starting out, fixing some issues will help familiarize yourself with the process and expectations, making things go more smoothly. If you'd like to tackle something without a mentor, the expectations will be higher on what you'll need to do on your own.