Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package workspaces #13947

Merged
merged 3 commits into from
Jul 26, 2024
Merged

Package workspaces #13947

merged 3 commits into from
Jul 26, 2024

Conversation

jneem
Copy link
Contributor

@jneem jneem commented May 22, 2024

Adds support for packaging an entire workspace, even when there are dependencies between the crates. The generated packages should be identical to the ones produced by packaging and publishing the crates one-by-one in dependency order, but the main benefit of this PR is that the packages can be created and verified locally before anything is published.

The main mechanism is the one in #13926, where we create a temporary local registry that "overlays" the true registry. We "publish" the crates in the local registry, which enables lockfile generation and verification of the dependent crates.

This adds --registry and --index flags to cargo package. They act
much like the same arguments to cargo publish, except that of course
we are not actually publishing to the specified registry. Instead, these
arguments affect lock-file generation for intra-workspace dependencies:
when simultaneously packaging a crate and one of its dependencies, the
lock-file will be generated under the assumption that the dependency
will be published to the specified registry.

You can also publish a subset of a workspace using -p arguments. In this case, there will be an error unless the chosen subset contains all of the dependencies of everything in the subset.

Fixes #10948. Based on #13926.

Compatibility issue

This PR introduces a case where cargo package will fail where it did not before: if you have a workspace containing two packages, main and dep@0.1.0, where main depends on dep@0.1.0 and dep@0.1.0 is already published in crates.io then attempting to package the whole workspace will fail. To be specific, it will package dep@0.1.0 successfully and then fail when trying to package main because it will see the two different packages for dep@0.1.0. Note that cargo publish will already fail in this scenario.

This shouldn't interfere with crates.io running cargo package for each package to diff the .crate files

  • This might interfere if someone tried to verify their "published" MSRV by running cargo package.

The failure could be avoided by changing the local overlay source to not error out if there's a duplicate package; see here. However, failing early has the advantage of catching errors early.

@rustbot
Copy link
Collaborator

rustbot commented May 22, 2024

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @weihanglo (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

  • @rustbot author: the review is finished, PR author should check the comments and take action accordingly
  • @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

@rustbot
Copy link
Collaborator

rustbot commented May 22, 2024

⚠️ Warning ⚠️

  • These commits modify submodules.

@rustbot rustbot added A-cli Area: Command-line interface, option parsing, etc. A-configuration Area: cargo config files and env vars A-dependency-resolution Area: dependency resolution and the resolver A-interacts-with-crates.io Area: interaction with registries A-manifest Area: Cargo.toml issues A-testing-cargo-itself Area: cargo's tests Command-package Command-publish S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 22, 2024
@rustbot rustbot added A-cli-help Area: built-in command-line help A-documenting-cargo-itself Area: Cargo's documentation labels May 22, 2024
@torhovland torhovland force-pushed the package-workspace branch 3 times, most recently from 0e1e2da to 140fc1d Compare May 23, 2024 09:40
@bors
Copy link
Collaborator

bors commented Jun 3, 2024

☔ The latest upstream changes (presumably #14004) made this pull request unmergeable. Please resolve the merge conflicts.

@jneem jneem force-pushed the package-workspace branch 3 times, most recently from 8cadf16 to 83b5ee5 Compare June 13, 2024 20:47
@jneem jneem marked this pull request as ready for review June 13, 2024 20:47
@jneem jneem changed the title [WIP] Package workspaces Package workspaces Jun 13, 2024
@jneem
Copy link
Contributor Author

jneem commented Jun 13, 2024

I think this is ready for a look.

There's a fair amount of churn in the tests: for workspace packaging we need to finish all the packaging before we start verifying, and this changes the order of console messages. I pushed as much of the test churn as I could into the first commit.

@jneem jneem force-pushed the package-workspace branch 2 times, most recently from 48c6a50 to c2d59e5 Compare June 13, 2024 23:13
tests/testsuite/package.rs Outdated Show resolved Hide resolved
jneem and others added 2 commits July 24, 2024 15:17
Co-Authored-By: Tor Hovland <55164+torhovland@users.noreply.github.com>
Co-Authored-By: Tor Hovland <55164+torhovland@users.noreply.github.com>
Takes local dependencies into account when packaging a workspace. Builds
a temporary package registry to provide local dependencies, and overlays
it on the upstream registry.

This adds `--registry` and `--index` flags to `cargo package`. They act
much like the same arguments to `cargo publish`, except that of course
we are not actually publishing to the specified registry. Instead, these
arguments affect lock-file generation for intra-workspace dependencies:
when simultaneously packaging a crate and one of its dependencies, the
lock-file will be generated under the assumption that the dependency
will be published to the specified registry.

Co-Authored-By: Tor Hovland <55164+torhovland@users.noreply.github.com>
@jneem
Copy link
Contributor Author

jneem commented Jul 26, 2024

@epage I think the latest round of comments have all been addressed now.

@epage
Copy link
Contributor

epage commented Jul 26, 2024

@bors r+

@bors
Copy link
Collaborator

bors commented Jul 26, 2024

📌 Commit a2f0a9e has been approved by epage

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 26, 2024
@bors
Copy link
Collaborator

bors commented Jul 26, 2024

⌛ Testing commit a2f0a9e with merge b5d44db...

@bors
Copy link
Collaborator

bors commented Jul 26, 2024

☀️ Test successful - checks-actions
Approved by: epage
Pushing b5d44db to master...

@bors bors merged commit b5d44db into rust-lang:master Jul 26, 2024
22 checks passed
bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 27, 2024
Update cargo

14 commits in 5f6b9a92201d78af75dc24f14662c3e2dacbbbe1..b5d44db1daf0469b227a6211b987162a39a54730
2024-07-19 18:09:17 +0000 to 2024-07-26 21:27:12 +0000
- Package workspaces (rust-lang/cargo#13947)
- test: migrate messages to snapbox (rust-lang/cargo#14242)
- chore: Update dependencies (rust-lang/cargo#14299)
- fix: remove rustc probe for `--check-cfg` support (rust-lang/cargo#14302)
- Misc test clean up (rust-lang/cargo#14297)
- Don't downgrade on prerelease `VersionReq` when update with --breaking. (rust-lang/cargo#14250)
- test: Migrate some json tests to snapbox (rust-lang/cargo#14293)
- Revert "fix: Ensure dep/feature activates the dependency on 2024" (rust-lang/cargo#14295)
- chore: bump cargo-test-support to 0.4.0 (rust-lang/cargo#14286)
- Bump to 0.83.0; update changelog (rust-lang/cargo#14285)
- Improved error message when `update --breaking` invalid spec. (rust-lang/cargo#14279)
- docs(test): Expand documentation of cargo-test-support (rust-lang/cargo#14272)
- test: Fix some test based on rustc version (rust-lang/cargo#14282)
- Use `Rc` instead of `Arc` for storing rustflags (rust-lang/cargo#14273)

r? ghost
bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 27, 2024
Update cargo

14 commits in 5f6b9a92201d78af75dc24f14662c3e2dacbbbe1..b5d44db1daf0469b227a6211b987162a39a54730
2024-07-19 18:09:17 +0000 to 2024-07-26 21:27:12 +0000
- Package workspaces (rust-lang/cargo#13947)
- test: migrate messages to snapbox (rust-lang/cargo#14242)
- chore: Update dependencies (rust-lang/cargo#14299)
- fix: remove rustc probe for `--check-cfg` support (rust-lang/cargo#14302)
- Misc test clean up (rust-lang/cargo#14297)
- Don't downgrade on prerelease `VersionReq` when update with --breaking. (rust-lang/cargo#14250)
- test: Migrate some json tests to snapbox (rust-lang/cargo#14293)
- Revert "fix: Ensure dep/feature activates the dependency on 2024" (rust-lang/cargo#14295)
- chore: bump cargo-test-support to 0.4.0 (rust-lang/cargo#14286)
- Bump to 0.83.0; update changelog (rust-lang/cargo#14285)
- Improved error message when `update --breaking` invalid spec. (rust-lang/cargo#14279)
- docs(test): Expand documentation of cargo-test-support (rust-lang/cargo#14272)
- test: Fix some test based on rustc version (rust-lang/cargo#14282)
- Use `Rc` instead of `Arc` for storing rustflags (rust-lang/cargo#14273)

r? ghost
bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 27, 2024
Update cargo

14 commits in 5f6b9a92201d78af75dc24f14662c3e2dacbbbe1..b5d44db1daf0469b227a6211b987162a39a54730
2024-07-19 18:09:17 +0000 to 2024-07-26 21:27:12 +0000
- Package workspaces (rust-lang/cargo#13947)
- test: migrate messages to snapbox (rust-lang/cargo#14242)
- chore: Update dependencies (rust-lang/cargo#14299)
- fix: remove rustc probe for `--check-cfg` support (rust-lang/cargo#14302)
- Misc test clean up (rust-lang/cargo#14297)
- Don't downgrade on prerelease `VersionReq` when update with --breaking. (rust-lang/cargo#14250)
- test: Migrate some json tests to snapbox (rust-lang/cargo#14293)
- Revert "fix: Ensure dep/feature activates the dependency on 2024" (rust-lang/cargo#14295)
- chore: bump cargo-test-support to 0.4.0 (rust-lang/cargo#14286)
- Bump to 0.83.0; update changelog (rust-lang/cargo#14285)
- Improved error message when `update --breaking` invalid spec. (rust-lang/cargo#14279)
- docs(test): Expand documentation of cargo-test-support (rust-lang/cargo#14272)
- test: Fix some test based on rustc version (rust-lang/cargo#14282)
- Use `Rc` instead of `Arc` for storing rustflags (rust-lang/cargo#14273)

r? ghost
@rustbot rustbot added this to the 1.82.0 milestone Jul 28, 2024
@weihanglo weihanglo mentioned this pull request Aug 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-cli Area: Command-line interface, option parsing, etc. A-cli-help Area: built-in command-line help A-completions Area: shell completions A-configuration Area: cargo config files and env vars A-dependency-resolution Area: dependency resolution and the resolver A-directory-source Area: directory sources (vendoring) A-documenting-cargo-itself Area: Cargo's documentation A-git Area: anything dealing with git A-interacts-with-crates.io Area: interaction with registries A-lockfile Area: Cargo.lock issues A-manifest Area: Cargo.toml issues A-registries Area: registries A-testing-cargo-itself Area: cargo's tests A-unstable Area: nightly unstable support Command-add Command-install Command-package Command-publish Command-uninstall Command-update S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cargo package --workspace is not very useful
5 participants