initial version of checksum based freshness #14137

Xaeroxe · 2024-06-25T05:18:22Z

Implementation for #14136 and resolves #6529

This PR implements the use of checksums in cargo fingerprints as an alternative to using mtimes. This is most useful on systems with poor mtime implementations.

This has a dependency on rust-lang/rust#126930. It's expected this will increase the time it takes to declare a build to be fresh. Still this loss in performance may be preferable to the issues the ecosystem has had with the use of mtimes for determining freshness.

rustbot · 2024-06-25T05:18:26Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @weihanglo (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

Cargo.toml

Add unstable support for outputting file checksums for use in cargo Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that `cargo` can read and use these values as an alternative to file mtimes. This PR powers the changes made in this cargo PR rust-lang/cargo#14137 Here's the tracking issue for the cargo feature rust-lang/cargo#14136.

bors · 2024-07-26T22:03:01Z

☔ The latest upstream changes (presumably #13947) made this pull request unmergeable. Please resolve the merge conflicts.

Xaeroxe · 2024-07-26T22:23:39Z

Merge conflicts resolved.

src/cargo/core/features.rs

src/cargo/util/command_prelude.rs

src/cargo/util/context/mod.rs

tests/testsuite/freshness_checksum.rs

src/cargo/core/compiler/fingerprint/mod.rs

… comments

This commit is not really necessary to be in the commit history. It is just for me to start over the test porting.

* cargo_env_changes * fingerprint_cleaner_does_not_rebuild * modify_only_some_files * rebuild_if_build_artifacts_move_forward_in_time * simulated_docker_deps_stay_cached * update_dependency_mtime_does_not_rebuild

These tests are modified or renamed to reflect the switch to checksum fingerprint: * bust_patched_dep * modifying_and_moving * rebuild_on_mid_build_file_modification * rebuild_sub_package_then_while_package * skip_mtime_check_in_selected_cargo_home_subdirs * use_mtime_cache_in_cargo_home

We don't rely on mtime anymore for checksum-based fingerprint

Two new tests: * checksum_actually_uses_checksum: chekcsum works when mtime forwards * same_size_different_content: checksum does check content

…yukang Add unstable support for outputting file checksums for use in cargo Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that `cargo` can read and use these values as an alternative to file mtimes. This PR powers the changes made in this cargo PR rust-lang/cargo#14137 Here's the tracking issue for the cargo feature rust-lang/cargo#14136.

Rollup merge of rust-lang#126930 - Xaeroxe:file-checksum-hint, r=chenyukang Add unstable support for outputting file checksums for use in cargo Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that `cargo` can read and use these values as an alternative to file mtimes. This PR powers the changes made in this cargo PR rust-lang/cargo#14137 Here's the tracking issue for the cargo feature rust-lang/cargo#14136.

Add unstable support for outputting file checksums for use in cargo Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that `cargo` can read and use these values as an alternative to file mtimes. This PR powers the changes made in this cargo PR rust-lang/cargo#14137 Here's the tracking issue for the cargo feature rust-lang/cargo#14136.

weihanglo · 2024-10-04T12:27:05Z

@bors try

Let's see how it really goes.

bors · 2024-10-04T12:28:25Z

⌛ Trying commit 4ddd017 with merge 5b99bb7...

initial version of checksum based freshness Implementation for #14136 and resolves #6529 This PR implements the use of checksums in cargo fingerprints as an alternative to using mtimes. This is most useful on systems with poor mtime implementations. This has a dependency on rust-lang/rust#126930. It's expected this will increase the time it takes to declare a build to be fresh. Still this loss in performance may be preferable to the issues the ecosystem has had with the use of mtimes for determining freshness.

bors · 2024-10-04T12:58:48Z

💔 Test failed - checks-actions

weihanglo · 2024-10-04T13:42:32Z

All tests are passed. I will do another round of review today.

weihanglo

@Xanewok

Would you be willing to rearrange commits and make them easier to review/track for future readers?

Some intermediate commits like a45d663 and 5b36d64 could be squashed, and some like what I've done for tests are helpful to understand they were copied from the other file.

If that sounds too much a burden, no worries. I can do a simple cleanup before this PR merges. Thank you :)

weihanglo · 2024-10-03T13:33:03Z

tests/testsuite/freshness_checksum.rs

+    let mut cmd = p.cargo("build").arg("-Zchecksum-freshness").build_command();
+    let output = cmd.output().unwrap();
+    assert!(
+        String::from_utf8(output.stderr)
+        .unwrap()
+        .contains("error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel")
+    );
+    assert!(!output.status.success());


We usually use Cargo-idiom test to write stuff, such as this one

Could you tweak it a bit to something like this?

Suggested change

let mut cmd = p.cargo("build").arg("-Zchecksum-freshness").build_command();

let output = cmd.output().unwrap();

assert!(

String::from_utf8(output.stderr)

.unwrap()

.contains("error: the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel")

);

assert!(!output.status.success());

p.cargo("build -Zchecksum-freshness")

.with_stderr_data(str![[

r#"[ERROR] the `-Z` flag is only accepted on the nightly channel of Cargo, but this is the `stable` channel

See https://doc.rust-lang.org/book/appendix-07-nightly-rust.html for more information about Rust release channels.

"#]])

.with_status(101)

.run();

weihanglo · 2024-10-03T13:49:16Z

src/cargo/core/compiler/fingerprint/mod.rs

+#[derive(Debug, Eq, PartialEq, Hash, Copy, Clone)]
+pub enum DepInfoPathType {


This isn't needed, right?

Suggested change

#[derive(Debug, Eq, PartialEq, Hash, Copy, Clone)]

pub enum DepInfoPathType {

enum DepInfoPathType {

weihanglo · 2024-10-03T14:02:34Z

src/cargo/core/compiler/fingerprint/mod.rs

+#[derive(Copy, Clone, Debug, Eq, PartialEq)]
+pub struct InvalidChecksumAlgo {}
+
+impl Display for InvalidChecksumAlgo {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "expected `sha256`, or `blake3`")
+    }
+}
+
+impl std::error::Error for InvalidChecksumAlgo {}


This type doesn't seem needed. We can inline the string message into InvalidChecksum::InvalidChecksumAlgo.

weihanglo · 2024-10-03T14:03:28Z

src/cargo/core/compiler/fingerprint/mod.rs

+    }
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq, thiserror::Error)]


Should be sufficient.

Suggested change

#[derive(Clone, Copy, Debug, Eq, PartialEq, thiserror::Error)]

#[derive(Debug, thiserror::Error)]

weihanglo · 2024-10-03T14:06:21Z

tests/testsuite/freshness_checksum.rs

Could we have a test checking that switch -Zfreshness-checksum on and off, and check they still build successfully?

weihanglo · 2024-10-04T19:06:03Z

tests/testsuite/freshness_checksum.rs

+#[cfg_attr(
+    not(all(target_arch = "x86_64", target_os = "windows", target_env = "msvc")),
+    ignore
+)]


This should be sufficient

Suggested change

#[cfg_attr(

not(all(target_arch = "x86_64", target_os = "windows", target_env = "msvc")),

ignore

)]

#[cfg(all(target_arch = "x86_64", target_os = "windows", target_env = "msvc"))]

weihanglo · 2024-10-04T19:25:03Z

tests/testsuite/dep_info.rs

Could we have a test verifying that the Cargo-flavored dep-info under fingerprint folder is correctly encoded? Something like relative_depinfo_paths_ws but simpler

rustbot assigned weihanglo Jun 25, 2024

This was referenced Jun 25, 2024

Tracking Issue for checksum freshness #14136

Open

Add unstable support for outputting file checksums for use in cargo rust-lang/rust#126930

Merged

Xaeroxe force-pushed the checksum-freshness branch from 320f73c to 310cd79 Compare June 25, 2024 05:37

tgross35 reviewed Jun 25, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

Xaeroxe force-pushed the checksum-freshness branch from cce62ba to 59441b6 Compare June 26, 2024 05:55

Xaeroxe mentioned this pull request Jul 1, 2024

MCP: Alternate cargo freshness algorithm, unstable flag to annotate depinfo file with checksums and file sizes rust-lang/compiler-team#765

Closed

3 tasks

rustbot added the A-infrastructure Area: infrastructure around the cargo repo, ci, releases, etc. label Jul 13, 2024

Xaeroxe force-pushed the checksum-freshness branch from 27e2a18 to c83be55 Compare July 13, 2024 14:58

Xaeroxe force-pushed the checksum-freshness branch from c83be55 to b8c21fa Compare July 26, 2024 22:23