Cargo should skip rerun-if-changed paths to files in published crate sources #11083
Comments
|
Ah - I've looked into things a bit more - Circle's caching does preserve mtimes, but I was only caching ~/.cargo/registry/cache but not ~/.cargo/registry/src. It seems like unifying the special casing of the src directory would still be nice here, but obviously not required! |
|
I've been hitting similar issue and decided to investigate a bit. I was only able to reproduce it with The easiest way to test this locally is to have one of those packages in dependencies (note: for libsqlite3-sys you need to enable I don't have a clue on how cargo works internally yet so my initial assumptions are: I also had an assumption about this behavior being triggered by pointing Additional note is that similar behavior happens if you have git-based dependencies and don't cache Until this is resolved it would probably be wise to warn people about this in the cargo book |
|
Your investigation is fantastic! Thank you for the thorough review :)
I believe that's because when Cargo walks the filesystem to collect mtimes, it also takes mtimes of directories into account. However, when cargo packages and uploads cargo/crates/cargo-util/src/paths.rs Line 224 in dd99005 And yep, to solve this, skipping |
|
@weihanglo thanks for the feedback, I already thought that I have a feeling that another way to address this would be to implement #6529 (
I also wonder what would be the impact of changing this part instead? it seems like the least invasive change since no tool should probably depend on the content of Anyhow, I'd be interested in trying to fix this myself once we decide on the approach |
|
A small update on this: I've started hacking around these approaches, changing mtime_recursive to ignore directories leads to an issue where cargo no longer detects if a file gets removed from the directory (the corresponding test also fails on cargo not detecting the creation of empty directory but I doubt it would've brought any issues on its own). Unless ignoring file removals and empty directory creations is acceptable I think it's indeed better to make |
|
At the same time, I can definitely see a scenario where someone wants to debug a registry crate by modifying the source in |
|
Have yet got time to figure out where the best place is for this change, but I created test for hacking on that. Click to expand the test#[cargo_test]
fn rerun_if_changed_directory() {
// build script of a dependency contains a `rerun-if-changed` pointing to a directory
//
// TODO: this currently asserts the buggy behavior described in rust-lang/cargo#11083
Package::new("mylib-sys", "1.0.0")
.file("mylib/balrog.c", "")
.file("src/lib.rs", "")
.file(
"build.rs",
r#"
fn main() {
// Changing to mylib/balrog.c will not trigger a rebuild
println!("cargo:rerun-if-changed=mylib");
}
"#,
)
.publish();
let p = project()
.file(
"Cargo.toml",
r#"
[package]
name = "foo"
version = "0.0.1"
[dependencies]
mylib-sys = "1.0.0"
"#,
)
.file("src/main.rs", "fn main() {}")
.build();
p.cargo("check").run();
// Delete regitry src to make directories being recreated with the latest timestamp.
cargo_home().join("registry/src").rm_rf();
p.cargo("check --verbose")
.with_stderr(
"\
[FRESH] mylib-sys v1.0.0
[FRESH] foo v0.0.1 ([CWD])
[FINISHED] dev [unoptimized + debuginfo] target(s) in [..]
",
)
.run();
}
Reiterate what was discussed on Zulip. #9455 was an attempt to making |
Problem
When running CI builds in CircleCI, I've noticed that caching the Cargo registry and target directory works most, but not all of the time to avoid rebuilding dependencies. In particular, tikv-jemalloc-sys is rebuilt on every CI run.
From what I can tell from reading the cargo::core::compiler::fingerprint log output, it looks like the issue is tikv-jemalloc-sys's build script emits a
rerun-if-changeddirective for the jemalloc source directory published with the crate. It seems like Circle's caching logic does not preserve file modification times, and while that seems to normally be fine it forces rebuilds in this one particular case.I believe the same should be true for basically any crate that does something similar (openssl-src, etc) but haven't checked.
Steps
rerun-if-changedon files bundled with the crate.cc, etc) are correctly detected as not needing a rebuild, but thererun-if-changedcrate does not.I've put together a simple project that reproduces the issue: https://github.com/sfackler/target-cache-test
A Circle build with fingerprint trace logging: https://app.circleci.com/pipelines/github/sfackler/target-cache-test/2/workflows/1dee508c-bd46-4729-a6a6-2715fff14c5a/jobs/2
A copy of the logs if you don't want to log into circle:
Possible Solution(s)
It seems like Cargo already assumes that the source of published crates will never change given that we don't see the same rebuilding behavior for most crates. It could potentially extend that to ignoring
rerun-if-changedpaths that point into the published crate source directory.Notes
No response
Version
No response
The text was updated successfully, but these errors were encountered: