This Development-cycle in Cargo: 1.90
This is a summary of what has been happening around Cargo development for the last 6 weeks which is approximately the merge window for Rust 1.90 as well as highlights from 1.87, 1.88, 1.89.
- Plugin of the cycle
- Implementation
- Design discussions
- All hands: XDG paths
- All hands: cargo linting
- All hands: doctests
- All hands: code-gen settings
- All hands:
cargo explain
cargo doc --serve
- Multi-line messages from build scripts
- Forgetting
cargo fmt
aftercargo fix
- Recursively find dependencies at a
path
- Include workspace license files with
cargo new
- Misc
- Focus areas without progress
Plugin of the cycle
Cargo can't be everything to everyone, if for no other reason than the compatibility guarantees it must uphold. Plugins play an important part of the Cargo ecosystem and we want to celebrate them.
Our plugin for this cycle is cargo-deny
,
a linter for Cargo.
A builtin linter for Cargo is being tracked in
#12235
along with ideas for lints.
Thanks to krpeid for the suggestion!
Please submit your suggestions for the next post.
Implementation
cargo fix
GSoC: Alternative design for cargo fix
will apply suggested fixes for lints
and has worked well for cleaning up sloppy code,
reducing the annoyance of toolchain upgrades where lints may change,
helping with Edition migrations,
and adopting of new lints in a code base.
However, cargo fix
- Can be slow (#13214)
- Only applies a subset of possible lints
- Can't be selective of which lints are fixed without a lot of mucking with
RUSTFLAGS
which was important with the 2024 Edition migration because some lints still had a lot of false positives at first
A problem with addressing these is the current architecture.
cargo fix
is implemented as a variant of cargo check
that replaces rustc
with cargo
being run in a special mode that will call rustc
in a loop, applying fixes until there are none.
While this special rustc
-proxy mode is running,
a cross-process lock is held to force only one build target to be fixed at a time to avoid race conditions.
This ensures correctness at the cost of performance and difficulty in making the rustc
-proxy interactive.
cargo-fixit
is a Proof of Concept for an alternative design,
developed by Pyr0de.
With this design,
cargo fixit
spawns cargo check
in a loop,
determining which build targets are safe to fix in this pass,
and applying the suggestions.
This puts the top-level program in charge of what fixes get applied,
making it easier to coordinate, allowing the locking to be removed and opening the door to an interactive mode.
This comes at the cost that fixes in packages lower in the dependency tree can cause later packages to rebuild multiple times,
slowing things down.
Regarding performance, cargo-fixit is showing promising results.
There remains
- Investigations into whether other optimizations are safe (cargo-fixit#52, cargo-fixit#21)
- Open questions on how the
cargo fix
interface should be maintained with this new design (stabilization issues) - Finish handling all of the remaining functionality
GSoC: Prototype a new set of Cargo "plumbing" commands
We've had a project goal for plumbing commands for a while and secona wrote cargo-plumbing as a prototype. The focus was on better understanding of what the plumbing commands can look like and what is needed from Cargo. Compromises had to be made in the actual result to not be blocked on what the Cargo Rust APIs currently allow (cargo-plumbing#82). For example, instead of solely relying on the manifests that the user passed in, the plumbing commands will re-read the manifests within each command, preventing callers from being able to edit them to get specific behavior out of Cargo, e.g. dropping all workspace members to allow resolving dependencies on a per-package basis.
cargo-plumbing
currently covers
locate-manifest
read-manifest
read-lockfile
lock-dependencies
write-lockfile
resolve-features
plan-build
GSoC: Build script delegation
Build scripts come at a compile time cost.
Even with cargo check
,
they must be built as if you ran cargo build
so they can be run.
While we need to identify ways to abstract common build script patterns
(#14948),
that may not always be doable.
However, if we can shift build scripts from being defined in every package that needs the functionality into a couple of core build script packages,
we can reduce the number of build scripts that need to be built and linked.
The first step in being able to delegate build scripts to packages is to be able to have multiple build scripts which is what namanlp has developed so far.
You can have
[]
= ["windows-manifest.rs", "release-info.rs"]
and your package can access their OUT_DIR
s as env!("windows-manifest_OUT_DIR")
and env!("release-info_OUTDIR")
(documentation).
The next phase is static parameters being defined in Cargo.toml
and then specifying dependencies using artifact-dependencies.
--target host
The following two commands are not equivalent:
$ cargo build
$ cargo build --target `rustc --print host-tuple`
While they both build for the host's platform tuple,
the first is run in "host mode"
while the second is run in "cross-compilation mode".
In "host mode",
build scripts and proc-macros are built the same as binaries and tests, including RUSTFLAGS
being applied, and everything is output to target/<profile>
.
In "cross-compilation mode" with the host's tuple,
everything still builds for the host's tuple but RUSTFLAGS
are not passed to build scripts and proc-macros and everything is output to target/<tuple>/<profile>
.
Some challenges with this:
- Naively, someone may run
cargo build
for packaging artifacts for their current platform andcargo --target <tuple>
for packaging artifacts for other platforms and the difference in the artifact-dir causes confusion and frustration in automation. - In some cross-compilation scenarios,
users also need to specify
RUSTFLAGS
for their build scripts and proc-macros. - When setting
build.target
to a specific platform tuple, there wouldn't be a way to set it back to the host without hard coding a specific host, whether host mode or cross-compilation mode
target-applies-to-host
and
host-config
are unstable features that have tried to address this.
In #13051,
kpreid
suggested a related feature to target-applies-to-host
,
a platform-tuple placeholder where the following mean the same thing:
$ cargo build --target host
$ cargo build --target `rustc --print host-tuple`
A user could then opt-in to always being in "cross-compilation" mode by setting in ~/.cargo/config.toml
:
[]
= "host"
When talking about this as a team, one concern was where all this could apply. For example, the following might not make sense:
[]
= "1"
We decided the alias would only exist for --target
/ build.target
for now.
We can evaluate where it makes sense to expand from there.
Another issue was whether host
would work as a placeholder.
As we'd be carving out host
from possible values,
we'd need to know whether that is safe.
We checked with T-compiler at #t-compiler > `--target=host` as alias to host triple in cargo @ 💬 and they were fine with it.
There is the question of whether host
is clear that this is a placeholder for the host tuple or if it could be confused with "host mode" and the unstable [host]
table for host builds.
One idea was to name it {host}
,
to make this an explicit variable substitution but weren't thrilled with that, especially on the command-line.
For now, we've settled on host-tuple
though the stabilization discussion is still on-going.
annotate-snippets
Update from 1.78
Muscraft released annotate-snippets 0.12 which includes
- swapping in rustc's renderer and fixing many bugs
- API re-design
This release is in preparation for replacing rustc's renderer with annotate-snippets. At this point, it appears that annotate-snippets is flexible enough and complete enough to handle all of rustc's needs and the remaining issues are with rustc itself and the adapter between rustc and annotate-snippets.
With annotate-snippets v0.12 out, epage started the effort to switch Cargo's user messages over to it which is being tracked in #15944.
Design discussions
All hands: XDG paths
Cargo stores all of its user-wide content under CARGO_HOME
, usually ~/.cargo
,
including configuration, caches, and locally built binaries.
There has been a long standing request to use the OS native paths for these,
particularly XDG Base Directories
(#1734).
While it can be difficult to take an existing application and migrate data to a new location without breaking compatibility, what makes this more difficult includes:
- Different versions of Cargo are expected to run side-by-side
rustup
integrates withCARGO_HOME
and is versioned independent of Cargo and supports interacting with every version of Cargo- They both use
$CARGO_HOME/bin
for installing binaries - Rustup sets up
$CARGO_HOME/env
for users to source for adding$CARGO_HOME/bin
to theirPATH
to access both Rustup and Cargo installed binaries - Rustup cleans up rustup and cargo content, including
$CARGO_HOME/bin
on uninstall - Cargo needs to be able to find third-party subcommands installed by both Cargo and Rustup
- Rustup's
cargo
proxy always setsCARGO_HOME
on the realcargo
- They both use
Working through the Cargo / Rustup interactions has been a problem for moving this forward (including the existing Phase 1 Pre-RFC) and we were able to get the Cargo and Rustup teams together at the All Hands to talk through these problems.
The most immediate problem is Rustup setting CARGO_HOME
.
To not break compatibility,
Cargo should always respect it if CARGO_HOME
is set.
However, if Rustup always sets it,
then Cargo will never use OS native paths.
The intention behind Rustup setting CARGO_HOME
is to make sure Rustup and Cargo use the same CARGO_HOME
for bin/
and env
.
This was particularly a problem at some point in the past
when they did not agree on a definition.
Because new versions of Rustup work with old versions of Cargo,
this remains an issue.
Both teams suspect it will be acceptable at this point
for Rustup to stop setting CARGO_HOME
.
However, we first need to characterize their diverging definitions of CARGO_HOME
and see if there is anything we can do to mitigate user problems.
This is being tracked in rustup#4502.
As Cargo then moves to OS native paths,
does Rustup need to continue to match behavior and use the same paths?
This boiled down to whether Rustup should continue to cleanup CARGO_HOME/bin
, caches, and config on uninstall.
If a user sets ~/.local/bin
as their install path,
Rustup could end up deleting user binaries.
In fact, Rustup does it today because some applications have chosen to install their binaries into ~/.cargo/bin
,
even if they weren't installed by cargo install
.
It's also likely not a good idea to be deleting the users configuration.
We decided that Rustup should only remove content it manages
(rustup#285).
With that resolved, we can consider on its own merits whether to use the same paths. We leaned towards them being configured separately but first we need to test the user experience for this to see how well it will work in practice. We should at least coordinate on whether to use application or XDG paths on Mac.
As for $CARGO_HOME/.env
, our expectation is that it will contain the default bin path for both Rustup and Cargo.
The rest of the mechanics are more program specific. We talked a bit about Cargo's transition. We still plan to do this in two phases, (1) make the more granular paths configurable and then (2) change the defaults to OS native paths. The change to OS native paths might not even need to happen in one go so long as we've solve the policy questions (particularly for Mac). For config, Cargo can read from both locations. For caches, Cargo can abandon the old location though there is a question of how to handle the cache garbage collection database, whether to have them be distinct or not. There is a question on how to transition the bin path.
All hands: cargo linting
Update from 1.83
In preparing for the All Hands, epage ran a vibe check for previously raised questions, including:
- What is the intent for the Rustc/Clippy divide and should Cargo mirror it?
- What namespace should Cargo lints live in?
- If Cargo share a namespace with Rustc or Clippy, should
RUSTFLAGS
affect cargo?- e.g. given
RUSTFLAGS=-Ddeprecated cargo check
, should Cargodeprecated
warnings also error?
- e.g. given
We mostly focused the discussion on RUSTFLAGS
behavior.
This would likely be implemented by Cargo passing the RUSTFLAGS
to Rustc and asking it to print the effective lint level for the lints.
This felt convoluted to the team and RUSTFLAGS
is intended as a low-level escape hatch and we should not be elevating its use in this way.
At the All Hands, the above questions were re-visited with members of the Cargo team, Compiler team, and Clippy team. While there is a quality divide between Rustc and Clippy, including performance, the general sentiment is to avoid the divide unless you absolutely have to.
One difference between Rustc and Clippy lints that was called out is that Clippy is more strict.
When a Clippy lint is uplifted into Rustc, the severity is lowered.
A lint that is a deny
for Clippy would be a warn
for Rustc.
The Clippy team also recommended adopting organizing lints by semantic groups that have the same lint level for the entire group, like Clippy's groups. This still leaves figuring out how this intersects with the lint levels having different meanings between Rustc and Clippy. If Cargo calibrates to Rustc's lint levels, should Cargo lower the severity of the groups or be more selective about what goes into each group?
When discussing the namespace for Cargo lints,
the recommendation is that Cargo only use the cargo::
namespace,
keeping the focus on the linter,
at the cost of requiring users to common lints in two different places,
even if it has extra configuration like unexpected_cfgs
.
Another reason for Cargo to focus on the cargo::
namespace is to make it easier for users to identify correctly where to open issues.
With these discussions out of the way, we can continue to work towards stabilizing Cargo's linter (#12235).
In preparation for that, we improved our tracking of new lints, including:
- Adopted a clippy-like template for requesting lints
- Created Issues for existing and requested Clippy Cargo lints
- Add an
A-new-lint
tag
All hands: doctests
Update from 1.82
When Cargo runs doctests,
it invokes rustdoc
against the library to build and run the tests.
This runs counter to the way the rest of Cargo works which creates warts in behavior, requiring special cases to workaround, and frustrating users
(e.g. a recent reddit thread).
Some examples of problems with doctests are:
- inability to run
cargo check
orcargo clippy
on them cargo test --workspace
rebuilding doctests when there was no change- cargo can't collect "unused depednency" messages from rustc to identify which dependencies are unused across all dev-dependencies
This also affects future plans including:
The Testing DevEx team has done some brainstorming on this problem (testing-devex-team#5), including:
- Building doctest support into the compiler,
parsing
#[doc]
attributes and generating#[test]
functions for them- Allows doctests on internal items and binary targets
- Doesn't allow for the "public interface" testing unless Rustc also links to the original lib or do some import path hackery
- Has issues with features like
compile_fail
and per-doctest Editions
- Using
rustdoc --output-format=doctest
to extract doctests, generate test files, build, and then run those (rust#134529)
epage took this to the Rustdoc team at the All Hands.
They had a variant of rustc having built-in knowledge of doctests:
run rustdoc
as an alternative compiler driver that will do the translation.
This idea was recorded in rust#141083.
One problem that came up later on the issue is cargo clippy
support as that is a separate compiler driver.
One idea is for the logic to be in rustc itself any drivers that link against it can provide a --doctest
flag like the --test
flag.
This is still left as an open area to further explore.
All hands: code-gen settings
The Cargo team discussed a request for exposing -Cforce-frame-pointers
in
[profiles]
(#15333).
This seemed like a reasonable idea.
We decided against changing the bench
profile to have this enabled
so that cargo bench
and cargo build --release
do the same thing by default.
This does tie into the broader question about deciding what compiler settings should be set in [profiles]
and if we can smooth out the process.
The burden for adding new settings across both rustc and cargo was highlighted by
kornelski
on
Internals
this is made harder by the fact that the Cargo team is generally disconnected from the development of these requests and it can be difficult to have the context for how these should be exposed in a higher-level, opinionated tool like Cargo.
The Cargo team followed up with the Compiler team at the All Hands.
An idea was discussed of a profile.*.codegen
field that would accept a list of
-C
code-gen settings
in a format Cargo could check against an allowlist, normalize, and de-duplicate.
Some -Z
flags, like Z fmt-debug
, may need to be turned into -Zunstable-options -C fmg-debug
to work with this.
The allowlist is important because Cargo's abstractions over rustc are intended to not be able to create unsound programs (e.g. having different floating point settings between compilation units) and to prevent arbitrary injection through response files. Updating the allowlist could be a step in the stabilization report and provide a sync point between the Compiler and Cargo team meetings which is currently lacking.
We then followed up on this in a Cargo team meeting as not everyone was at the All Hands or even at that session.
Unlike RUSTFLAGS
,
this is something Cargo can reasonably parse and validate.
However, both have the problem of potentially interacting poorly with
built-in Cargo behavior.
The allowlist could help with this as well.
We did not come to any definitive answer in the discussion.
cargo explain
All hands: At the All Hands,
timclicks attended an open time with the Cargo team to discuss ways to improve the experience with compiler diagnostics.
The core of the idea is to provide one place for users to go to get more detailed information on an error code or lint name.
Currently, you can view error code descriptions with rustc --explain
or cargo --explain
and clippy lints with cargo clippy --explain
.
There isn't a local view for rustc lints.
These views dump markdown to the screen without any styling.
timclicks proposed there to be a cargo explain
subcommand that could take in an error code or lint name and stylize the markdown output.
There is the question of how information for third-party lints could be found once rust#66079 is stabilized.
This could be well served by having a built-in pager
(#11213).
This led epage to floating an idea that he has been considering for a bit: supporting a compilation watch mode
(#9339)
but in the style of bacon.
Combined with this idea of cargo explain
,
maybe it could be an interactive mode (e.g. cargo check -i
)
that allowed you to select a diagnostic and have it pop up an "explain" view.
In response, someone brought up that the compiler knows more than is currently shown and maybe this could be included in the json output so that the user could drill further into a diagnostic in interactive mode.
Like with many other subcommands, this would likely best be experimented with as a
custom subcommand
before integrating it into Cargo.
cargo doc --serve
The Cargo team reviewed an old requested for cargo doc
to get a --serve
flag
(#4966).
The given use case is for web browsers that limit plugin access to file://
URLs.
This also seems relevant for
WSL,
dev containers,
and remote development.
However, we would be concerned about anyone using this in production.
If the cargo doc
process is staying alive to serve files,
it seems a natural extension to rebuild the documentation on changes
(#9339).
That can then lead to considerations for live reloading
or broader ideas of watch functionality as an "interactive mode"
(example)
as discussed earlier.
Presenting serve/watch as an "interactive mode" can help set expectations
when it comes to production use.
A challenge with this is the support burden this could put on the Cargo team,
which includes supporting users through unmet expectations and feature creep.
For example, mdbook has several issues open related to its serve
functionality.
rustup doc
also has similar needs and it would be good for this effort to be coordinated with them
(
rustup#2151,
rustup#3112
).
Possible paths forward include experimenting with this as a nightly-only feature or creating a third-party subcommand to experiment with the full blown interactive mode.
Multi-line messages from build scripts
kornelski and ratmice started back up the conversation on multi-line build script errors and warnings.
Build scripts communicate to cargo through directives
of the format cargo::<name>=<params>\n
,
including cargo::warning=<message>\n
and cargo::error=<message>\n
.
The only way to report multiple lines of text for the same message is to split the message across multiple of these directives,
causing each line to render as error: <name>@<ver>: <line>
.
epage wrote up a
summary,
got input, and brought this before the Cargo team.
The options includes:
# Concatenation
cargo::error=header
cargo::error+=second line
# Block start/end
cargo::error_start=
header
second line
cargo::error_end=
# Line-count prefix
cargo::error_start=2
header
second line
# Block start/end + line prefix
cargo::error_start=
cargo::error=header
cargo::error=second line
cargo::error_end=
When the team stepped through the options, we determined that our values for a solution include:
- Every line having a
cargo::
prefix to match with our previous communication that cargo ignores everything else - Explicit end-of-message, rather than being inferred
- Some felt uncomfortable with a blank line being sufficient to end buffering
- Some felt uncomfortable with cargo::error and cargo::warning changing their semantics from non-buffered to buffered
- Graceful handling of nested content or multi-threaded environments
These values excluded "block start/end" and "line-count prefix" solutions. "Concatenation" and "block start/end + line prefix" have points both for and against them. In the end, we favored the value of "explicit end-of-message" and preferred "block start/end + line prefix".
There are still open questions, though we didn't feel strongly whether those needed to be resolved before implementation or during the stabilization process.
cargo fmt
after cargo fix
Forgetting cargo fix
and cargo clippy --fix
will apply suggested fixes from lints.
However, the output is not always pretty and making it so would be difficult.
If you are like me, it's easy to not notice that cargo fmt
is needed until you see CI fail.
It would help if cargo fix
also handled formatting
(#12608).
One problem is users may not be using cargo fmt
and having cargo fix
apply it for the first time could be unwanted on its own and hide the actual fix that went in.
A way to reduce unwanted edits is if we had an interface to tell cargo fmt
to only re-format the parts changed by cargo fix
.
The exact interaction for these would have to be carefully considered and may still produce unwanted reformatting.
At the next level, Cargo could have a .cargo/config.toml
field
[fix]
rustfmt = true
If we did this, we may want to have Cargo run cargo fmt --check
first,
similar to our "VCS dirty" check.
At a minimum, cargo fix
could recommend running cargo fmt
if changes were made.
Our discussion was inconclusive and we moved it to Internals.
path
Recursively find dependencies at a Cargo will automatically find a package within its git repo, like:
[]
= { = "https://github.com/bevyengine/bevy.git" }
However, the same can't be said for path
dependencies where an exact path is needed.
Say you cloned Bevy into /Users/epage/Development/bevy
, you'd have to depend on it as:
[]
= { = "/Users/epage/Development/bevy/crates/bevy_math/" }
For git
, the recursive path search has run into several issues, including:
- A performance hit for walking the filesystem on every run and parsing the manifests (#14395)
- Ambiguous package names (#11858)
Non-determinism in package selection when multiple share a name, fixed in #14239)Warning about ambiguous package names even when not referenced (#10752), fixed in #14239- Non-fatal error messages for unreferenced packages (#13724)
At minimum, we can improve the error message from:
error: no matching package named `definitely_not_bar` found
location searched: /Users/eric/Temp/foo/crates/bar
required by package `foo v0.1.0 (/Users/eric/Temp/foo)`
to something like:
error: no matching package named `definitely_not_bar` found at `bar/`
note: required by package `foo v0.1.0 (/Users/eric/Temp/foo)`
help: package `definitely_not_bar` exists at `bar/definitely_not_bar/`
Due to the problems we've had with git
,
we were hesitant to extend the recursive path search to path
.
This likely wouldn't be reconsidered until at least
#14395
is addressed.
We talked about the idea of caching the relative path inside of the Cargo.lock
.
If the package is no longer at that path (update of git
dependency, user edited the layout at path
),
Cargo would re-scan for the package.
The big hurdle would be plumbing this advisory information from one lockfile, through the resolve, to the next lockfile.
cargo new
Include workspace license files with When you run cargo new
in a workspace,
it will automatically inherit workspace.package
fields
and the workspace.lints
table.
Commonly, license files need to be copied into a package for distribution via cargo publish
and it would help if symlinked any license files Cargo found in the workspace root into the package
(13328).
On a mechanics side,
there are downsides to symlinking on Windows that
could make it a bad policy by default,
regardless of the platform cargo new
is running on.
Copying, always or as a fallback to symlinking failing,
has its own drawbacks.
We could have a workspace.new-files
that cargo new
will copy in.
It's hard to evaluate a solution like this without better understanding
where we might go with templateing (#5151).
We could have a package.extra-files
which cargo publish
will copy into the .crate
file like it does package.readme
when it's outside of the package root.
If we made it this general,
we'd need to also allow users to specify where the files would go.
There is package.license-file
which cargo publish
will already copy into the .crate
file if it is outside the package root, like with package.readme
.
However, only one file can be specified (#5933)
because this is intended for a custom license when package.license
is insufficient.
See also
#8537,
#9908,
#9972.
We vaguely discussed a way to map license names,
including a custom license identifier,
to specific files.
cargo publish
would then copy these into the package root inside the .crate
file.
We didn't end up reaching any specific conclusions.
Misc
- ranger-ross is experimenting with new
build-dir
layouts in #15947 while working to improve how we test it in #15874
Focus areas without progress
These are areas of interest for Cargo team members with no reportable progress for this development-cycle.
Project goals in need of owners
Ready-to-develop:
Planning:
- Disabling of default features
- RFC #3416:
features
metadata- RFC #3487: visibility (visibility)
- RFC #3486: deprecation
- Unstable features
- Pre-RFC: Global, mutually exclusive features
- RFC #3553: Cargo SBOM Fragment
How you can help
If you have ideas for improving cargo, we recommend first checking our backlog and then exploring the idea on Internals.
If there is a particular issue that you are wanting resolved that wasn't discussed here, some steps you can take to help move it along include:
- Summarizing the existing conversation (example:
Better support for docker layer caching,
Change in
Cargo.lock
policy, MSRV-aware resolver ) - Document prior art from other ecosystems so we can build on the work others have done and make something familiar to users, where it makes sense
- Document related problems and solutions within Cargo so we see if we are solving to the right layer of abstraction
- Building on those posts, propose a solution that takes into account the above information and cargo's compatibility requirements (example)
We are available to help mentor people for S-accepted issues on zulip and you can talk to us in real-time during Contributor Office Hours. If you are looking to help with one of the bigger projects mentioned here and are just starting out, fixing some issues will help familiarize yourself with the process and expectations, making things go more smoothly. If you'd like to tackle something without a mentor, the expectations will be higher on what you'll need to do on your own.