Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local registry overlays #13926

Merged
merged 3 commits into from
Jun 11, 2024
Merged

Add local registry overlays #13926

merged 3 commits into from
Jun 11, 2024

Conversation

jneem
Copy link
Contributor

@jneem jneem commented May 17, 2024

This PR adds (private to cargo internals) support for local registry overlays, in which you can locally pretend to add packages to remote registries; the local packages will have the same source ids as the remote registry that you're overlaying.

There are two ways to set up these overlays: programmatically using GlobalContext::local_overlays and through the __CARGO_TEST_PACKAGE_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS environment variable. You can't set up these overlays with .cargo/config.

The motivation for this is packaging workspaces. When we're packing a workspace, we'd like to be able to pretend (for lockfile generation and verification) that some workspace packages are already published even though they aren't.

@rustbot
Copy link
Collaborator

rustbot commented May 17, 2024

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

  • @rustbot author: the review is finished, PR author should check the comments and take action accordingly
  • @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

@rustbot rustbot added A-configuration Area: cargo config files and env vars A-testing-cargo-itself Area: cargo's tests S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 17, 2024
if let Some(ref mut p) = self.process_builder {
let env_value = format!("{}={}", url, path);
p.env(
"__CARGO_TEST_PACKAGE_CONFUSION_VULNERABILITY_DO_NOT_USE_THIS",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The term of art for this kind of vulnerability is "Dependency Confusion". As with so many names in computer science, it's better to have consistency then trying to find a better name. This term was coined in the blog post https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610 which might be worth linking in comments about how users could easily miss use this.

@Eh2406
Copy link
Contributor

Eh2406 commented May 20, 2024

Catching Eric up on some of the context, Arlo and I discussed the questions from #10948 (comment) at office hours on Thursday. Adding this API, exclusively for internal use, seemed like the least hacky solution.

Cargo.toml Outdated Show resolved Hide resolved
src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved
src/cargo/sources/overlay.rs Show resolved Hide resolved
src/cargo/util/context/mod.rs Outdated Show resolved Hide resolved
src/cargo/sources/overlay.rs Outdated Show resolved Hide resolved
@jneem jneem mentioned this pull request May 22, 2024
@jneem jneem requested a review from Eh2406 May 23, 2024 15:54
@jneem
Copy link
Contributor Author

jneem commented May 23, 2024

Thanks for the review! I think this is ready for another look -- we managed to successfully use this for #10948 as discussed.

/// affect any newly created `PackageRegistry`s.
///
/// See [`crate::sources::overlay::OverlayConfusionAttack`] for why you shouldn't use this.
pub fn local_overlays(&self) -> RefMut<'_, HashMap<SourceId, PathBuf>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd hope we can avoid putting this on GlobalContext but its hard to say without reviewing the follow up at the same time. Should we defer this to the follow up PR?

My motivation is that it gets messy adding and removing from the Global Context and if we can directly instantiate things, the easier it will be to maintain.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can give a little more context before we decide whether or not to defer. cargo package needs the overlay to take effect during a call to ops::compile. AFAICT there are three possibilities:

  • put the overlay configuration in GlobalContext
  • put the overlay configuration in the ephemeral workspace that cargo package creates
  • have cargo package do something lower-level than ops::compile

The workspace option has the advantage that the overlay configuration doesn't need to be removed (because the workspace is discarded). It seems like it would just require a little work in resolve_ws and resolve_ws_with_opts to get the overlay configuration out of the workspace and insert it into the PackageRegistry

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workspace seems the most beneficial. Whichever way we go, still seems like it could be split out into a follow up PR

src/cargo/sources/config.rs Outdated Show resolved Hide resolved
src/cargo/sources/config.rs Outdated Show resolved Hide resolved
Cargo.toml Outdated Show resolved Hide resolved
src/cargo/sources/config.rs Outdated Show resolved Hide resolved
src/cargo/sources/config.rs Outdated Show resolved Hide resolved
@epage
Copy link
Contributor

epage commented Jun 7, 2024

btw feel free to clean up your commit history for how you want to present it for review and merge.

This adds a new mechanism for overlaying sources. An overlayed source
returns packages from two (or more) sources. This functionality is not
intended for public use, but it will be useful for packaging a workspace
that contains inter-crate dependencies.

Co-authored-by: Tor Hovland <55164+torhovland@users.noreply.github.com>
@jneem jneem force-pushed the overlay branch 2 times, most recently from c99add9 to beb929e Compare June 10, 2024 20:19
src/cargo/ops/resolve.rs Outdated Show resolved Hide resolved
src/cargo/ops/resolve.rs Outdated Show resolved Hide resolved
Adds workspace configuration options (not user-exposed) for overlaying
sources.
@rustbot rustbot added A-infrastructure Area: infrastructure around the cargo repo, ci, releases, etc. Command-add Command-package Command-update labels Jun 11, 2024
@epage
Copy link
Contributor

epage commented Jun 11, 2024

Thanks!

@bors r+

@bors
Copy link
Collaborator

bors commented Jun 11, 2024

📌 Commit ba9dd1e has been approved by epage

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 11, 2024
@bors
Copy link
Collaborator

bors commented Jun 11, 2024

⌛ Testing commit ba9dd1e with merge 4dcbca1...

@bors
Copy link
Collaborator

bors commented Jun 11, 2024

☀️ Test successful - checks-actions
Approved by: epage
Pushing 4dcbca1 to master...

@bors bors merged commit 4dcbca1 into rust-lang:master Jun 11, 2024
22 checks passed
bors added a commit to rust-lang-ci/rust that referenced this pull request Jun 12, 2024
Update cargo

14 commits in b1feb75d062444e2cee8b3d2aaa95309d65e9ccd..4dcbca118ab7f9ffac4728004c983754bc6a04ff
2024-06-07 20:16:17 +0000 to 2024-06-11 16:27:02 +0000
- Add local registry overlays (rust-lang/cargo#13926)
- docs(change): Don't mention non-existent workspace.badges (rust-lang/cargo#14042)
- test: migrate binary_name to snapbox (rust-lang/cargo#14041)
- Bump to 0.82.0; update changelog (rust-lang/cargo#14040)
- tests: Migrate alt_registry to snapbox (rust-lang/cargo#14031)
- fix: proc-macro example from dep no longer affects feature resolution (rust-lang/cargo#13892)
- chore: Bump cargo-util-schemas to 0.5 (rust-lang/cargo#14038)
- chore(deps): update rust crate pulldown-cmark to 0.11.0 (rust-lang/cargo#14037)
- fix: remove `__CARGO_GITOXIDE_DISABLE_LIST_FILES` env var (rust-lang/cargo#14036)
- chore(deps): update rust crate itertools to 0.13.0 (rust-lang/cargo#13998)
- fix(toml): remove `lib.plugin` key support and make it warning (rust-lang/cargo#13902)
- chore(deps): update compatible (rust-lang/cargo#13995)
- fix: using `--release/debug` and `--profile` together becomes an error (rust-lang/cargo#13971)
- fix(toml): Convert warnings that `licence` and `readme` files do not exist into errors (rust-lang/cargo#13921)

r? ghost
@rustbot rustbot added this to the 1.81.0 milestone Jun 12, 2024
bors added a commit that referenced this pull request Jul 26, 2024
Package workspaces

Adds support for packaging an entire workspace, even when there are dependencies between the crates. The generated packages should be identical to the ones produced by packaging and publishing the crates one-by-one in dependency order, but the main benefit of this PR is that the packages can be created and verified locally before anything is published.

The main mechanism is the one in #13926, where we create a temporary local registry that "overlays" the true registry. We "publish" the crates in the local registry, which enables lockfile generation and verification of the dependent crates.

This adds `--registry` and `--index` flags to `cargo package`. They act
much like the same arguments to `cargo publish`, except that of course
we are not actually publishing to the specified registry. Instead, these
arguments affect lock-file generation for intra-workspace dependencies:
when simultaneously packaging a crate and one of its dependencies, the
lock-file will be generated under the assumption that the dependency
will be published to the specified registry.

You can also publish a subset of a workspace using `-p` arguments. In this case, there will be an error unless the chosen subset contains all of the dependencies of everything in the subset.

Fixes #10948. Based on #13926.

### Compatibility issue

This PR introduces a case where `cargo package` will fail where it did not before: if you have a workspace containing two packages, `main` and `dep@0.1.0`, where `main` depends on `dep@0.1.0` and `dep@0.1.0` is already published in crates.io then attempting to package the whole workspace will fail. To be specific, it will package `dep@0.1.0` successfully and then fail when trying to package `main` because it will see the two different packages for `dep@0.1.0`. Note that `cargo publish` will already fail in this scenario.

This shouldn't interfere with crates.io running `cargo package` for each package to diff the `.crate` files
- This might interfere if someone tried to verify their "published" MSRV by running `cargo package`.

The failure could be avoided by changing the local overlay source to not error out if there's a duplicate package; see [here](#13926 (comment)). However, failing early has the advantage of catching errors early.
@weihanglo weihanglo mentioned this pull request Aug 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-configuration Area: cargo config files and env vars A-infrastructure Area: infrastructure around the cargo repo, ci, releases, etc. A-registries Area: registries A-testing-cargo-itself Area: cargo's tests A-workspaces Area: workspaces Command-add Command-package Command-update S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants