Removed use of iteration through a HashMap/HashSet in rustc_incremental and replaced with IndexMap/IndexSet #110040

ndrewxie · 2023-04-07T06:35:35Z

This allows for the #[allow(rustc::potential_query_instability)] in rustc_incremental to be removed, moving towards fixing #84447 (although a LOT more modules have to be changed to fully resolve it). Only HashMaps/HashSets that are being iterated through have been modified (although many structs and traits outside of rustc_incremental had to be modified as well, as they had fields/methods that involved a HashMap/HashSet that would be iterated through)

I'm making a PR for just 1 module changed to test for performance regressions and such, for future changes I'll either edit this PR to reflect additional modules being converted, or batch multiple modules of changes together and make a PR for each group of modules.

rustbot · 2023-04-07T06:35:43Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @lcnr (or someone else) soon.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

ndrewxie · 2023-04-07T07:01:05Z

Aah it appears some tests are failing, ig I didn't hit all the test cases on my local machine while testing (more specifically, the cranelift and gcc backend). I'll make a few edits, then retry tomorrow

cuviper · 2023-04-07T19:20:39Z

Note that IndexMap iterator order is only deterministic with respect to the insertion order, so you must also ensure that all insertions and removals are also deterministic to get the kind of stability that rustc wants. Or you can take steps like sorting the map after that's done, so iteration will follow the sort order.

(I'm not saying this PR is necessarily a problem in that way, just noting it for caution.)

ndrewxie · 2023-04-07T19:58:23Z

Aah, I see. After I fix the issue with cranelift build failing, I'll go investigate that, and keep it in mind for future changes as well :)

rustbot · 2023-04-07T19:59:04Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo

ndrewxie · 2023-04-07T20:00:41Z

Also just a quick question - I failed the build test for codegen_cranelift and codegen_gcc because the tests I was doing on my local machine never hit those modules. Is there a way I can test those as well?

For codegen_cranelift, I found using --target=i686-pc-windows-gnu worked, but is there an equivalent target I can use to make sure my code doesn't break codegen_gcc?

Thanks!

bjorn3 · 2023-04-07T20:22:24Z

./x.py check should check both cg_clif and cg_gcc. ./x.py build will only build them if they are listed in the codegen-backends option of config.toml.

ndrewxie · 2023-04-07T20:23:46Z

./x.py check should check both cg_clif and cg_gcc. ./x.py build will only build them if they are listed in the codegen-backends option of config.toml.

ohhhh I see, I haven't really been using ./x.py check... maybe I should start doing so, my ~~tabletop jet engine~~ laptop would certainly thank me

Noratrieb · 2023-04-07T22:24:19Z

@bors try @rust-timer queue

bors · 2023-04-07T22:24:28Z

⌛ Trying commit 916c1ca501f3e134b8cd8e61201d5a01a9b19d12 with merge e709771db4bde2ba3203fc1eb0b315b71b35c8c6...

bors · 2023-04-08T00:08:24Z

☀️ Try build successful - checks-actions
Build commit: e709771db4bde2ba3203fc1eb0b315b71b35c8c6 (e709771db4bde2ba3203fc1eb0b315b71b35c8c6)

rust-timer · 2023-04-08T01:26:49Z

Finished benchmarking commit (e709771db4bde2ba3203fc1eb0b315b71b35c8c6): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.7%	[2.5%, 2.9%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

lcnr · 2023-04-12T09:23:13Z

with rust-lang/compiler-team#533 we should be able to instead use https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/unord/struct.UnordMap.html for a lot of these i think? 🤔

not sure what's the current status here but it feels preferable to use these whenever possible, cc @michaelwoerister

ndrewxie · 2023-04-12T18:08:04Z

with rust-lang/compiler-team#533 we should be able to instead use https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/unord/struct.UnordMap.html for a lot of these i think? 🤔

Ooo, that's a pretty cool data structure, didn't know that existed! Yeah, I guess that provides an even better determinism guarantee than FxIndexMap, because we either have to work with it in an order-agnostic way, or sort it. IDK if it'll cause performance regressions from extra .sorts() (in places where iteration by Fn instead of FnMut isn't sufficient), but I'll be happy to migrate over (whenever possible) and run those performance tests if michaelwoerister confirms that it's indeed the best practice :)

michaelwoerister · 2023-04-19T08:36:56Z

IndexMap and UnordMap can both make things deterministic but have slightly different properties:

IndexMap makes sure that iteration order matches insertion order. That means that one has to be careful to insert things in a deterministic way. E.g. the result of some_regular_hashmap.iter().collect::<IndexMap<_, _>>() will still not be deterministic, and it's not always easy to see if insertion order is deterministic.
UnordMap "erases" the iteration order. It is safe to feed things into it in any order one wants. It's API allows to do iteration-order-independent operations because they are safe to do (and often that's all that's needed). It also allows to extract its items into an ordered collection, but only in a deterministic way. However, this can be expensive operation since we have to sort the items in a way that is stable across compilation sessions.

I've been trying to use UnordMap where possible because it is a bit more robust (i.e. it introduces a kind of indeterminism firewall). But for maps where it's trivially obvious that they are built deterministically, IndexMap is a good choice too.

lcnr · 2023-04-24T13:36:14Z

@ndrewxie do you want to try to move some of these uses to UnordMap? please retag this to S-waiting-to-review whenever you're happy. I think it's fine to use IndexMap in places where it's not stricty necessary but would prefer to replace as much as possible with UnordMap

@rustbot author

ndrewxie · 2023-04-26T01:51:49Z

Alright, will do 👍

ndrewxie · 2023-06-05T04:56:02Z

@rustbot label -S-waiting-on-author +S-waiting-on-review

lcnr

I have to admit that the bool argument to into_sorted_stable_ord is very confusing to me and even after looking it up in the documentation I always have to wait a second and think about whether it is correct.

I would like us to either add a const CAN_USE_UNSTABLE_SORT: bool to StableOrd or alternatively always use a stable sort and only add unstable sorting where it is actually performance critical.

Please either do that in this PR or I will open a github issue once your PR lands.

Other thoughts: UnordItems using into() via From instead of having a method called collect is also confusing imo.

after dealing with review comments:

r=me,michaelwoerister

compiler/rustc_incremental/src/persist/fs.rs

lcnr · 2023-06-06T12:41:22Z

compiler/rustc_incremental/src/persist/work_product.rs

+        .items()
+        .map(|(_, path)| path.as_str())
+        .into_sorted_stable_ord(false)


&String should implement StableOrd

Hmm, apparently not. Would it be correct to impl StableOrd for &T if T is StableOrd? I believe it should, but just checking in (if not, then I'll just add the impl for &String on its own)

Would it be correct to impl StableOrd for &T if T is StableOrd?

yes

compiler/rustc_interface/src/queries.rs

michaelwoerister · 2023-06-07T13:55:24Z

I would like us to either add a const CAN_USE_UNSTABLE_SORT: bool to StableOrd ...

@lcnr, that sounds like a great idea! If you open an issue, feel free to assign it to me.

… few misc issues, added collect to UnordItems

rustbot · 2023-06-08T04:39:20Z

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

…fixed a few misc issues, added collect to UnordItems

ndrewxie · 2023-06-08T04:44:34Z

Alright, I applied the review suggestions and switched UnordItems::into_sorted_stable_ord to use CAN_USE_UNSTABLE_SORT, and added collect to UnordItems

@rustbot label -S-waiting-on-author +S-waiting-on-review

michaelwoerister · 2023-06-08T07:22:22Z

Thanks, @ndrewxie -- looks good!

@bors r=lcnr,michaelwoerister

bors · 2023-06-08T07:22:24Z

📌 Commit 3f324a8 has been approved by lcnr,michaelwoerister

It is now in the queue for this repository.

bors · 2023-06-08T07:30:07Z

⌛ Testing commit 3f324a8 with merge a0df04c...

lcnr

a few nits for code which can be cleaned up now. Feel free to do them in a separate PR once this has landed

compiler/rustc_incremental/src/persist/dirty_clean.rs

compiler/rustc_incremental/src/persist/fs.rs

compiler/rustc_query_system/src/dep_graph/dep_node.rs

bors · 2023-06-08T10:31:45Z

☀️ Test successful - checks-actions
Approved by: lcnr,michaelwoerister
Pushing a0df04c to master...

rust-timer · 2023-06-08T12:05:00Z

Finished benchmarking commit (a0df04c): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.0%	[0.5%, 3.5%]	2
Regressions ❌ (secondary)	1.3%	[1.3%, 1.3%]	1
Improvements ✅ (primary)	-3.3%	[-6.1%, -0.6%]	2
Improvements ✅ (secondary)	-2.5%	[-2.5%, -2.5%]	1
All ❌✅ (primary)	-0.6%	[-6.1%, 3.5%]	4

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.9%	[-4.2%, -1.0%]	7
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.9%	[-4.2%, -1.0%]	7

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 647.603s -> 648.385s (0.12%)

…compiler-errors Removed unnecessary &String -> &str, now that &String implements StableOrd as well Applied a few nits suggested by lcnr to PR rust-lang#110040 (nits can be found [here](rust-lang#110040 (review)).) Making a new PR because the old one was already merged, and given that this just applies changes that were already suggested, reviewing it should be fairly open-and-shut.

rustbot assigned lcnr Apr 7, 2023

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 7, 2023

This comment has been minimized.

Sign in to view

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 8, 2023

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 24, 2023

This comment has been minimized.

Sign in to view

ndrewxie force-pushed the issue-84447-partial-1 branch from 69ba7dc to f5f638c Compare June 5, 2023 04:17

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 5, 2023

lcnr reviewed Jun 6, 2023

View reviewed changes

lcnr added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 6, 2023

Removed stable/unstable sort arg from into_sorted_stable_ord, fixed a…

54d7b32

… few misc issues, added collect to UnordItems

fixup! Removed stable/unstable sort arg from into_sorted_stable_ord, …

4ae250b

…fixed a few misc issues, added collect to UnordItems

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 8, 2023

This comment has been minimized.

Sign in to view

Whoops, submodule change was actually valid - undoing fixup

3f324a8

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 8, 2023

lcnr reviewed Jun 8, 2023

View reviewed changes

bors added the merged-by-bors This PR was explicitly merged by bors. label Jun 8, 2023

bors merged commit a0df04c into rust-lang:master Jun 8, 2023

rustbot added this to the 1.72.0 milestone Jun 8, 2023

ndrewxie mentioned this pull request Jun 12, 2023

Removed unnecessary &String -> &str, now that &String implements StableOrd as well #112538

Merged

celinval mentioned this pull request Jun 21, 2023

Upgrade rust toolchain to nightly-2023-06-20 model-checking/kani#2551

Merged

4 tasks

Removed use of iteration through a HashMap/HashSet in rustc_incremental and replaced with IndexMap/IndexSet #110040

Removed use of iteration through a HashMap/HashSet in rustc_incremental and replaced with IndexMap/IndexSet #110040

Conversation

ndrewxie commented Apr 7, 2023 • edited Loading

rustbot commented Apr 7, 2023

ndrewxie commented Apr 7, 2023 • edited Loading

This comment has been minimized.

cuviper commented Apr 7, 2023

ndrewxie commented Apr 7, 2023

rustbot commented Apr 7, 2023

ndrewxie commented Apr 7, 2023

bjorn3 commented Apr 7, 2023

ndrewxie commented Apr 7, 2023 • edited Loading

Noratrieb commented Apr 7, 2023

This comment has been minimized.

bors commented Apr 7, 2023

bors commented Apr 8, 2023

This comment has been minimized.

rust-timer commented Apr 8, 2023

Overall result: no relevant changes - no action needed

lcnr commented Apr 12, 2023

ndrewxie commented Apr 12, 2023

michaelwoerister commented Apr 19, 2023

lcnr commented Apr 24, 2023

ndrewxie commented Apr 26, 2023

This comment has been minimized.

ndrewxie commented Jun 5, 2023

lcnr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndrewxie Jun 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelwoerister commented Jun 7, 2023

rustbot commented Jun 8, 2023

ndrewxie commented Jun 8, 2023 • edited Loading

This comment has been minimized.

michaelwoerister commented Jun 8, 2023

bors commented Jun 8, 2023

bors commented Jun 8, 2023

lcnr left a comment

Choose a reason for hiding this comment

bors commented Jun 8, 2023

rust-timer commented Jun 8, 2023

Overall result: no relevant changes - no action needed

ndrewxie commented Apr 7, 2023 •

edited

Loading

ndrewxie commented Apr 7, 2023 •

edited

Loading

ndrewxie commented Apr 7, 2023 •

edited

Loading

ndrewxie Jun 8, 2023 •

edited

Loading

ndrewxie commented Jun 8, 2023 •

edited

Loading