Tracking Issue for incr. comp. red/green testing

[michaelwoerister]

We already have quite a bit of testing in place for incremental compilation but some of it needs to be adapted to the new red/green change tracking system. This issue will track the needed changes and the progress we make on them.

Incremental Compilation Testing Strategy

We use four major kinds of tests in order to keep regressions at bay:

1. Synthetic end-to-end tests which are small, synthetic test cases that -- given a certain change to the source code -- test that the expected set of object files gets re-compiled. These tests use the #![rustc_partition_reused]/#![rustc_partition_translated] attributes and are located in src/test/incremental.
2. Synthetic tests of input and intermediate-result fingerprints. The incr. comp. system computes fingerprints (hashes) of various things and then uses these fingerprints to check if something has changed in comparison to the previous compilation session. The test cases in src/test/incremental/hashes test, at a fine-grained level, that various changes in the source code lead to changed fingerprints of various intermediate results. They use the #[rustc_clean]/#[rustc_dirty] attributes to indicate which things are expected to have changed and which are not. incr.comp.: Create Test Case for Incr. Comp. Hash for enums #36674 is an example issue with a description of how these tests work.
3. Synthetic dependency graph based tests. These tests check that certain paths exist or do not exist in the dependency graph. The dependency graph records which values had to be accessed (transitively) in order to compute a certain value during the compilation process. By checking for expected and unexpected paths we can make sure that dependency tracking doesn't miss anything (=anything that we actively test for) and that the compiler doesn't unnecessarily access data it doesn't need (which leads to unnecessary cache invalidation). These tests use the #[rustc_if_this_changed]/#[rustc_then_this_would_need] and located in src/test/compile-fail
4. Real-world test cases that take code bases of existing Rust projects, replay their git history, compile each commit and at every step make sure the compiling with and without incr-comp-cache produces the same binaries. These tests are based on the cargo-incremental tool and can be found at https://github.com/rust-icci and https://travis-ci.org/rust-icci. They have proven to be very useful for finding holes in the tracking system that are not covered by any of the synthetic test cases yet.

Changes Needed for Red/Green

Most of the test cases can stay the same also with the new tracking system:

• The synthetic end-to-end tests still make sense and are expected to be updated as part of the regular implementation.
• The synthetic dependency graph based tests can stay as they are. The new tracking system still uses a dependency graph underneath and that dependency graph has the same semantics as in the old system.
• The real-world test cases are black-box tests and are not affected by compiler internals.

There are some things, however, that should be extended and improved for the red/green system:

• In the old tracking system, we only had fingerprints for the HIR/AST nodes which represent the "input" to the compilation pipeline. With the red/green system we compute fingerprints of almost all intermediate results the compiler produces. The tests in src/test/incremental/hashes should therefore be updated to:
  • test many more intermediate results, and
  • be adapted to be based on actual fingerprints instead of on presence of dependency graph nodes after invalidation.
• In the final red/green system fingerprints for some "green" nodes will not be re-computed as they can just be copied from the previous session. It would make sense to implement a special compiler mode that does recompute the fingerprint of such nodes and asserts that the recomputed fingerprint is the same as the one from the previous compilation.

Action Items

• Change src/test/incremental/hashes tests to being genuinely fingerprint-based once incr.comp.: Move Metadata-hash loading to DepGraph. #44702 and incr.comp.: Add new DepGraph implementation. #44772 are resolved/merged.
  UPDATE: This bullet point is completely implemented once incr.comp.: Switch to red/green change tracking, remove legacy system. #44901 has landed.
  UPDATE 2: I just verified that the current master branch already contains all needed functionality.
• Adapt the tests in src/test/incremental/hashes to check additional fingerprints. The main issue for this is incr.comp.: Update fingerprint-based auto tests for red/green tracking. #44924. This is ready to tackle.
• Implement a "validate all fingerprints" compiler mode that validates fingerprints loaded from the previous session (implemented in incr.comp.: Verify stability of incr. comp. hashes and clean up various other things. #45867).

Please leave comments below if you think something's missing or if you have any remarks!

cc @nikomatsakis @vitiral

[vitiral]

Thanks for adding this! Let me say how excited I am that all this work from the previous iteration of incr-comp is still usable. These tests are extremely extensive, excellent work!

Some comments (I will edit this post as I review):

• The incr hash tests should have a link to this. It's a good guide of them.

[vitiral]

@michaelwoerister I'll take call_expressions.rs to get my feet wet but it looks like we are blocked on a few issues.

Can you have a Status: existing tests [run/don't run] at the top so new contributors can know when they can jump in with a passing suite?

[michaelwoerister]

Can you have a Status: existing tests [run/don't run] at the top so new contributors can know when they can jump in with a passing suite?

Good idea. I'll add this tomorrow (it's already getting late in my time zone). Hopefully these kinds of tests will be updateable by the end of next week.

[vitiral]

It would be good to also provide a checkboxed list of what is blocking us as well :)

[michaelwoerister]

All sub-tasks are done!