Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Per-version hashing information #1126

Closed
Mark-Simulacrum opened this issue Dec 20, 2021 · 1 comment
Closed

Per-version hashing information #1126

Mark-Simulacrum opened this issue Dec 20, 2021 · 1 comment

Comments

@Mark-Simulacrum
Copy link
Member

Since the merge of rust-lang/rust#89836, we are seeing high variance on the regression-31157-opt benchmark:

image

This is likely due to hashing being effectively randomized, affecting some hash table layout (I've not narrowed down where/what exactly).

I suspect this may be creating noise on other benchmarks, as well. Our significance algorithm should learn to cope with this in another ~20 commits (roughly 2-3 days) as our backwards window is for 50 commits, but it likely makes sense to try to avoid this problem at the root, and in general it'll make regression triage harder by increasing the baseline changes commit to commit even with small code deltas.

I am not sure yet whether a revert of rust-lang/rust#89836 makes sense, or whether we should use RUSTC_FORCE_INCR_COMP_ARTIFACT_HEADER on rustc-perf to reduce noise levels... in the meantime I'll set the env variable for perf and that'll hopefully eliminate noise in the meantime.

cc @jyn514

@Mark-Simulacrum
Copy link
Member Author

This is essentially fixed by #1127.

image
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant