Allow specifying dependencies for individual artifacts

[dmarcuse]

This RFC proposes adding a way to specify dependencies (in a cargo manifest) that are only used for a single artifact (lib/bin/etc). This is intended to allow crate authors publishing multiple artifacts in a single crate to avoid pulling in dependencies unnecessarily, improving compile times and binary size.

Previous discussion

Rendered

[alexreg]

@ehuss I've been reviewing @dmarcuse's efforts on this for a while now (she's a new contributor), and would like to say it's in a great state to get some official feedback. I believe you're the appropriate person here, so if you could have a look and give any feedback, that would be much appreciated!

[djc]

It would be great to have this fixed!

I'm a bit concerned about the proposed solution direction of adding separate per-target dependency tables -- as the RFC mentions, it would make it harder to get a good understanding of the different dependencies involved. Instead I would propose a new per-target features (name to be bikeshedded) key that lists the features that this target enables.

(I believe the second unresolved question would be solved with the new resolver that @ehuss is working on, starting from rust-lang/cargo#7820.)

[alexcrichton]

One of the stickler points in the past about this feature has been what to do about the crates.io index. The index is where Cargo learns dependency information for crates.io-sourced packages, and the dependencies (even optional/target-specific ones) are currently in the index. A relatively large change that this would naively require would be to update the index with all this new information. In any case I think it's important for the RFC to address the situation about the index.

I have an idea, however, which means that the index doesn't have to get updated (which is always a tricky thing to do). The main "target" that the crates.io index cares about is the library itself, not any binary, example, or test suite. The binaries and examples only come into play with cargo install. That is today that if we didn't do anything to the index, I don't believe we'd break anything currently using the index except cargo install. Typically Cargo usage would continue to work.

To fix cargo install I think we can have Cargo change how it implements cargo install. Currently it does resolution all on the index itself, but instead of that I think we can have Cargo download the crate file, read the full Cargo.toml to learn about all these new dependencies, and then proceed as usual.

Put another way, I think we can get away with not updating the index at all here and continue to function normally. In discussing this with other Cargo folks the main thing that we'd lose is a high-fidelity view into reverse dependencies because the index/crates.io would not be informed of bin-specific dependencies. We could try to fix that with extra JSON information on publish which doesn't go into the index, only crates.io's database, but it's altogether not the worst downside!

[alexreg]

Instead I would propose a new per-target features (name to be bikeshedded) key that lists the features that this target enables.

I personally don't like the indirectness of this solution. It would be nice to allow binaries to enabled features as well, however. Like how downstream crates can.

[alexreg]

@alexcrichton Sounds like a nice approach that doesn't lose too much in honesty. I guess it all depends how painful it would end up being to update the index, which I really don't know... but your solution as I understand is not what I'd called "hacky", at least! :-)

[dmarcuse]

I'm not familiar with how the crates.io index works, but that sounds like a good workaround to me. Would it negatively impact build times when using cargo install?

[ehuss]

One concern about not being in the index is that future changes like rust-lang/cargo#4316 would be more difficult (not a blocker, just something to note).

It might be worth adding Swift to the "Prior Art" section, and taking a look at how they approached it. Just a few days ago they released a new version that included essentially the same thing this RFC proposes. You can read more about it at https://github.com/apple/swift-evolution/blob/master/proposals/0226-package-manager-target-based-dep-resolution.md. One interesting thing about their approach is that each target lists the dependencies it uses by name, but keeps the actual dependency declarations in one place. I think that would be more similar to the idea of having a set of default features for a target that could enable optional dependencies.

I wouldn't completely dismiss the idea of using default target features instead. I think that would be substantially easier to implement, easier to integrate with tooling, would avoid the reverse-dependency problem, and be easier to document/explain. It would also make some scenarios easier to express. For example, if you have 3 binaries that depend on clap, you would only need to list features = ["clap"] in each one, instead of the entire dependency specification (like dependencies = {clap = {version = "2.33", features=["suggestions", "color"]}}, which can be awkward to repeat and update, and maybe not even possible with the "only specified once" rule?).
It adds a level of indirection, but the simplicity seems attractive. It also can cause rebuilds of a library if you switch between different binaries with different features (not sure if there is a way to mitigate that, but with incremental should be very fast).

[dmarcuse]

I agree that there's overlap with required-features, but I think this approach still has some benefits:

• Conceptually, I find it easier to understand - compilation artifacts are already relatively distinct units (being compiled from different source files) and in my opinion it's more intuitive to assign dependencies to these directly rather than defining optional dependencies, features, and then required features for each artifact.
• Less effort would be required by users to add dependencies to an artifact than defining new features in addition to optional dependencies
• In my mind, features are generally meant to be a way to customize behavior and functionality of a crate. Using features to assign dependencies to different artifacts further overloads this functionality, and defining features that effectively serve no purpose (e.g. a feature that enables required dependencies for a bin but has zero effect on the lib) seems inefficient and more confusing.

[djc]

I don't think anyone is arguing that there's overlap with required-features -- that field seems to be about a precondition for building a target, whereas this RFC is about dependencies of the target once the decision to build it has been made. Both use features, but for very different purposes.

While the bin build targets are distinct, cargo builds a single large dependency graph (with one root per target) to build all the targets for a particular build command invocation.

In the case of adding dependencies to a single build target, it is arguably a bit more work (but note that dependencies marked as optional = true implicitly introduce a feature of the same name). On the other hand, if you have a dependency like clap for more than one build target, the features approach would actually result in less work: for the second target, you just toggle on the feature rather than having to respecify the full details for the same dependency (and having to keep them in sync for the future).

It seems to me that features are already defined crate-wide, so the situation of features being defined for a lib that also influence a bin target is already happening anyway. Managing dependencies seems to me one of the primary applications of features in the first place, so reusing the same mechanism here would be sensible.

In any case, the features approach should be clearly explained in the alternatives section, with the attendant advantages and disadvantages.

[ehuss]

it's more intuitive to assign dependencies to these directly rather than defining optional dependencies, features, and then required features for each artifact.

To be clear, I wasn't advocating doing anything with required-features. Just adding a list of "enable these features when this target is built".

Less effort would be required by users to add dependencies to an artifact than defining new features in addition to optional dependencies

I think it will be more work if you have more than one target that wants to use the same dependency. Although for the single-use situation it is less work, the difference is pretty small. Compare:

[dependencies]
clap = {version="2", optional=true}

[[bin]]
name = "foo"
features = ["clap"]

to:

[[bin]]
name = "foo"
[bin.dependencies]
clap = {version="2"}

Mostly a difference of one line. I agree that conceptually it is not as straightforward in this simple case, but once you have just one more binary, it becomes worse.

And it becomes conceptually more complex for any tool that works with Cargo's dependencies. They will all need to be updated to handle target-specific dependencies, whereas with features most of them should work as-is.

It's also not clear to me what happens if multiple targets define dependencies on the same thing. Do they have to be exactly the same? If there are differences, how are those handled?

defining features that effectively serve no purpose … seems inefficient and more confusing.

How would it be inefficient? I believe it would be essentially zero cost.

[alexreg]

From my perspective, having bin targets enabling features would be something of an abuse of the purpose of features. As far as I know (and intuitively), they're meant to be externally-facing boolean parameters that change the feature set (and only as a by-product the dependencies) of a particulate crate that is getting used. Sometimes a bin target would conceptually want to enable a feature for the crate lib, because that's the nature of the binary, but sometimes it just needs a few extra dependencies. That would entail creating a dummy feature, which would itself be usable by downstream crates (not intended), and would require a dummy name which is referenced in bin.dependencies as a dependent feature. By all means, allow bin targets to depend on features, but I think the fundamental point of this RFC is to allow them to have dependencies directly, for the above reasons and because it's so much more natural and intuitive to reason about.

As for the "cost", I don't think there's anything more than the most minute cost for the more indirect feature-based approach, but I would hazard a guess that @dmarcuse meant "inefficient" in terms of characters in the Cargo.toml file or perhaps "conceptually" inefficient, akin to what I just described (correct me if I'm wrong).

[dmarcuse]

Yep, @alexreg is exactly right about what I was trying to say.

[kornelski]

I find it annoying when I run cargo build --bin foo and Cargo refuses, because I haven't enabled the required features.
To me it's an unnecessary error and artificial self-inflicted problem: Cargo knows what I mean, and could do what I mean, but refuses on principle.

So I'd like to see a more lenient handling of required-features anyway.

[kornelski]

Cargo is somewhat undecided about what should be in scope of a single crate, and what should be a workspace.

For example, Cargo is supposed to discourage users from having both lib and bin targets in a single crate: rust-lang/api-guidelines#167

And this feature goes in the other direction: helps having more targets in a single crate.

So maybe this feature should not exist?

• It would be a clear guideline that if you have targets that need different dependencies, then it's a sign that you're supposed to use separate crates in a workspace.

• Avoids overlap with required-features. For basic cases like lib+bin crate needing clap, fixing the usability issue of enabling required-features when building the bin would be an 80/20 solution.

• Avoids complicating the crates index. Currently the index is super valuable for analysis of the ecosystem. It's way easier to use one git repo than to download all tarballs of all versions of all crates.

• If multiple diverse targets are needed to work uniformly, another approach could be to improve workspace handling:

  • maybe allow cross-referencing crates in a workspace "my examples are provided by these other crates" (this is already an informal pattern)
  • Improve UX of bulk-publishing workspaces to crates.io. The hassle of publishing multiple crates is usually the main motivator for cramming more into one crate.

[dmarcuse]

@kornelski you bring up some good points, but I disagree with your perspective that separate crates in a workspace should be used here. Using separate crates does allow you to fully separate the dependency trees in a way that's already supported by cargo, but it introduces its own complexities:

• There's no way to make a symbol accessible to workspace members only, meaning there's no way to avoid publicly exporting a symbol that's needed by a binary depending on a library.
• Because there are now two separate crates, the orphan rule kicks in, making it impossible to implement foreign traits for types from the library in the binary. For a specific example, I was working on a project where I had several types defined in the library, and wanted to implement postgres_types::FromSql for them in the binary, and instead had to write wrapper types and implementations, duplicating a significant chunk of the code.

[0x5c]

This definitely solves the issue this RFC is targeting, and we can always add dependencies as a better UX alternative later.

Doesn't changing the enabled features trigger a full rebuild of the lib target?

[tae-soo-kim]

Overall, this is a very good RFC addressing a problem since 2015. I'm alongside @dmarcuse advocating a direct per-crate dependency list, avoiding using features for this kind of dependency specification. The reasons are pretty much as explained by @dmarcuse and others: Features are indirect, require more efforts, and don't play nicely with editors. (Cargo) features are meant to provide (program) features. Specifying dependencies is often not a feature. The fact that my bin crate uses clap but my lib crate doesn't is not a feature per se, and shouldn't bother introducing a feature for this sole purpose.

Issues have been filed repeatedly within these years, rust-lang/rust#95513 and rust-lang/cargo#12854 and some others. I don't think there's a nice solution without per-crate dependencies. Playing with features on this is going astray. The fundamental problem is cargo compiles each crate on its own thus unifying dependencies of different crates simply doesn't make sense. While there could be a common dependency list for all crates, it shall not remove the ability to specify dependencies for each crate.

However, one problem I see in this RFC itself is its terminology. For example, I guess when it says:

Allow specifying dependencies that are only used for one build artifact ([lib], [[bin]], etc) in a single crate.

It actually means:

Allow specifying dependencies that are only used for one crate ([lib], [[bin]], etc) in a single package.

This RFC universally substituted crate for package and artifact for crate, which I think is a mistake.

[wiiznokes]

I don't see it on the RFC so I thought i will ask but does

[bin.dependencies.myproject_cli]
clap = "*"

and

[bin.dependencies.myproject_cli.clap]
version = "*"

would work ? Or maybe this syntax is induced by that of the RFC (I am not a toml expert)

[wiiznokes]

I summarized all the above mentioned concerns with the proposal below:

• If you have more than one target that wants to use the same dependency, it is more work

Why not use the optional syntax for this concern ?

[wiiznokes]

Also, I wanted to share another use case for this i haven't saw. It would be to use only some portion of code of a package, without including his dependencies. I don't think you could to that without this rfc.

[epage]

@rfcbot close

#3374 was created as an alternative to this but has since been closed. A major problem that had that is also shared with this is that it takes an existing problem for users (feature unification) and applies it not just to the package level but to the individual build target (see #3374 (comment)). Adding new dependencies tables would also cause a significant ecosystem churn that we need to be mindful of.

Note also that this isn't the only problem in dealing with wanting a package to scale to complex build targets needs. We discussed this as a team. For myself, I think we should focus more on improving the experience with workflows before we re-evaluate how much more we should do for improving complex package needs.

[rfcbot]

Team member @epage has proposed to close this. The next step is review by the rest of the tagged team members:

• @Eh2406
• @Muscraft
• @arlosi
• @ehuss
• @epage
• @joshtriplett
• @weihanglo

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[JarredAllen]

I'd have liked this RFC or something like it such that, if I have a binary and a library in the same crate, where the binary requires clap and the libary does not, I can (maybe with setting some configuration in the Cargo.toml or other config files) use:

cargo build --lib

to build just the library without pulling clap into the dependency tree, or

cargo build --bin <bin_name>

to build the binary with clap as a dependency.

The current required-features behavior doesn't meet that, since I have to know all the required features and explicitly provide them (which can be hard to remember and type, since there might be a bunch of them) or let cargo give me an error, copy the list of required features and then try again with those features pasted in (which takes extra steps).

With this RFC being closed, is there any path towards that behavior being possible that's being pursued?

[djc]

@JarredAllen did you already read the discussion referenced in #2887 (comment)?

[rfcbot]

The final comment period, with a disposition to close, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This is now closed.