x.py's naming of stages is confusing

[nnethercote]

Rust has the standard multi-stage structure that all bootstrapping compilers have.

A newcomer who knows about compiler stages will be confident with this, until they run a command like ./x.py build --stage 1 and get output like this:

Building stage0 std artifacts 
    ...
Copying stage0 std from stage0
Building stage0 test artifacts 
    ...
Copying stage0 test from stage0
Building stage0 compiler artifacts 
    ...
Copying stage0 rustc from stage0
Building stage0 codegen artifacts 
    ...
Assembling stage1 compiler 
Building stage1 std artifacts 
    ...
Copying stage1 std from stage1
Building stage1 test artifacts 
    ...
Copying stage1 test from stage1
Building stage1 compiler artifacts 
    ...
Copying stage1 rustc from stage1
Building stage1 codegen artifacts 
    ...
Building rustdoc for stage1 
    ...

For a newcomer, this is completely bizarre.

• Why is it building stage 0? Isn't stage 0 the compiler you download?
• How does it assemble the stage 1 compiler before building the stage 1 artifacts?
• Why is it building two stages? Shouldn't it only build one stage?

The key to understanding this is that x.py uses a surprising naming convention:

• A "stage N artifact" is an artifact that is produced by the stage N compiler.
• The "stage (N+1) compiler" is assembled from "stage N artifacts".
• A --stage N flag means build with stage N.

Somebody had to explain this to me when I started working on Rust. I have since had to explain it to multiple newcomers. Even though I understand it now, I still find it very confusing, and I have to think carefully about it all.

Here's a naming convention that makes more sense to me:

• A "stage N artifact" is an artifact that is produced by the stage (N-1) compiler.
• The "stage N compiler" is assembled from "stage N artifacts".
• A --stage N flag means build stage N, using stage (N-1).

That way, a command like ./x.py build --stage 1 would produce output like this:

Building stage1 std artifacts 
    ...
Copying stage1 std from stage1
Building stage1 test artifacts 
    ...
Copying stage1 test from stage1
Building stage1 compiler artifacts 
    ...
Copying stage1 rustc from stage1
Building stage1 codegen artifacts 
    ...
Assembling stage1 compiler 

Is there any appetite for this change? I realize it would be invasive and people would have to update their aliases, build scripts, etc. But it might be worthwhile to avoid the ongoing confusion to both newcomers and experienced contributors. Deprecating the word "stage" in favour of "phase" could help with the transition.

[steveklabnik]

FWIW I also found this confusing for a while, and would support this chnage.

…

On Apr 10, 2019, at 7:23 PM, Nicholas Nethercote ***@***.***> wrote: Rust has the standard multi-stage structure that all bootstrapping compilers have. A newcomer who knows about compiler stages will be confident with this, until they run a command like ./x.py build --stage 1 and get output like this: Building stage0 std artifacts ... Copying stage0 std from stage0 Building stage0 test artifacts ... Copying stage0 test from stage0 Building stage0 compiler artifacts ... Copying stage0 rustc from stage0 Building stage0 codegen artifacts ... Assembling stage1 compiler Building stage1 std artifacts ... Copying stage1 std from stage1 Building stage1 test artifacts ... Copying stage1 test from stage1 Building stage1 compiler artifacts ... Copying stage1 rustc from stage1 Building stage1 codegen artifacts ... Building rustdoc for stage1 ... For a newcomer, this is completely bizarre. Why is it building stage 0? Isn't stage 0 the compiler you download? Why is it building two stages? Shouldn't it only build one stage? The key to understanding this is that x.py uses a surprising naming convention: A "stage N artifact" is an artifact that is produced by the stage N compiler. The "stage (N+1) compiler" is assembled from "stage N artifacts". A --stage N flag means build with stage N. Somebody had to explain this to me when I started working on Rust. I have since had to explain it to multiple newcomers. Even though I understand it now, I still find it very confusing, and I have to think carefully about it all. Here's a naming convention that makes more sense to me: A "stage N artifact" is an artifact that is produced by the stage (N-1) compiler. The "stage N compiler" is assembled from "stage N artifacts". A --stage N flag means build stage N, using stage (N-1). That way, a command like ./x.py build --stage 1 would produce output like this: Building stage1 std artifacts ... Copying stage1 std from stage1 Building stage1 test artifacts ... Copying stage1 test from stage1 Building stage1 compiler artifacts ... Copying stage1 rustc from stage1 Building stage1 codegen artifacts ... Assembling stage1 compiler Is there any appetite for this change? I realize it would be invasive and people would have to update their aliases, build scripts, etc. But it might be worthwhile to avoid the ongoing confusion to both newcomers and experienced contributors. Deprecating the word "stage" in favour of "phase" could help with the transition. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ehuss]

I think this is essentially #57963. I agree it is confusing (I even drew a little picture to help me understand it).

[cuviper]

@ehuss that diagram is excellent!

[cuviper]

cc @Mark-Simulacrum @alexcrichton -- related to the discussion we had at all-hands.

[Mark-Simulacrum]

My plan is to put in sometime this weekend to figure out and propose PRs for the changes discussed at the all hands (which should, to an extent, remove this issue by making dealing with --stage largely unnecessary). We may separately want to land a patch that changes stage "naming" though.

[Manishearth]

I basically have to re-figure this out any time I'm doing anything complex with rustc, and yeah, we should change the naming here.

A key issue here is that the rust compiler is actually a thin wrapper around underlying libraries, which are shipped with rust like the stdlib is. So which stage is which differs based on whether you care about the stdlib or the compiler. Kind of. I've complained about this before and generally seen agreement that this needs fixing but not as many good solutions

[spastorino]

@nnethercote I think your explanation and @ehuss diagram are perfect for rustc-guides 😄. I also would like to see this change happen, meanwhile can we document this on Rustc? are you comfortable if your explanation and diagrams are reused for rustc-guides?.
@igaray was interested in getting started with rustc-guides documentation maybe it's something he can use as a way to start.

[unbalancedparentheses]

Friday was the first time I compiled rust and it was pretty confusing for me too. I really like the diagram.

[nikomatsakis]

At minimum, we should move that content to the rustc-guide, yes. I am agnostic about the names. I'm wary that all names will be confusing unless there is a picture, and if there is a picture, any name may be fine.

[Mark-Simulacrum]

The current plan (discussed and mentioned at all hands) is to work towards not needing to say "--stage" unless you're hacking on rustbuild itself or doing something odd; I'm not sure how quickly we can make that happen, but hopefully quite soon.

[scottmcm]

@Mark-Simulacrum Does that apply to --keep-stage as well? That's the one I use most, since it's currently essentially if you're doing anything in core or std...

[Mark-Simulacrum]

Yes, keep-stage as well -- you can see some notes here: https://paper.dropbox.com/doc/Rustbuild-All-Hands-2019--AbYzEwK9y9KN~OKiOg2bMnWrAQ-MLKsxxj1SXORp4Qg8aGbQ

[nnethercote]

are you comfortable if your explanation and diagrams are reused for rustc-guides?.

Absolutely, please use the above words any way you like.

[RalfJung]

FWIW I suffered the exact same confusion until @eddyb explained the rustc stage naming to me.

[robinmoussu]

Same for me. Does this means that if I only need to build the sources, I only need to pass --stage 0 and not --stage 1? I am currently hacking on rustdoc.

[eddyb]

@robinmoussu The best way to get the correct behavior is to run tests, IME.
./x.py test --stage 1 src/test/rustdoc{,-ui} will build the compiler and rustdoc exactly once and run its tests. If you want to hack on rustdoc without the compiler being built, but instead relying on the rustc-dev component from a nightly/master build, then you need @Mark-Simulacrum's plan which should allow that (last mentioned on this internals post).

[eddyb]

I think overall the worst UX aspect is that ./x.py build --stage 1 doesn't do the same thing as ./x.py build --stage 1 src/libstd but rather it does almost everything that
./x.py build --stage 2 src/libstd does.

And in general ./x.py build --stage N, is useless, since it's missing host libraries (I'm not even sure you can use Xargo or Cargo's -Z build-std without them).

So making src/libstd the default and having to write ./x.py build --stage N src/rustc to get the (less useful) compiler binary with no libraries, seems adequate to me.

Oh and we need to get rid of fulldeps tests and similar things: ./x.py test --stage 1 should not build the compiler twice, but it does (for fulldeps tests and a few other things).
Or we could keep them and run them at stage 2 (ideally without building the compiler thrice).

Right now I'm forced to use this command:
./x.py test --no-fail-fast --stage 1 src/test/{rustdoc,rustdoc-ui,mir-opt,codegen,codegen-units,incremental,pretty,debuginfo,compile-fail,run-make,run-fail,ui}
in order to make sure I'm not missing out on any important tests, without doing unnecessary builds.

[robinmoussu]

Thanks.

And I agree this mess need to be clean-up. I hope someone will someday do the work.

[eddyb]

Would it also help if the sysroot directories were named stage{1,2}-sysroot instead of just stage{1,2}? It would also match stage0-sysroot nicely.

Then some intuition there is that stage1-sysroot has executables (and their runtime dependencies) from stage0-rustc but libraries from stage1-std (built by stage1-sysroot/bin/rustc aka stage0-rustc).

[RalfJung]

A --stage N flag means build with stage N.

To make things even more confusing, this is not actually always true... ./x.py build src/tools/miri --stage 1 builds with the stage 1 compiler artifacts, i.e., it uses the stage 2 compiler. How that fits into the rest of the scheme is beyond me.

[RalfJung]

@oli-obk pointed out that stage 0 Miri is being built by the bootstrap compiler, but against the stage 0 compiler artifacts (i.e., it is linked with those). 🤯

[jyn514]

I have a proposal for explaining this better in rust-lang/rustc-dev-guide#843. Please let me know if that helps at all and if not, what I could do to make it better :)

[jyn514]

I have not gotten a lot of feedback about rust-lang/rustc-dev-guide#857 except from new contributors. Do you think it explains this well enough to close the issue, or are there still parts of bootstrapping that are confusing? I'm happy to add more documentation if you think it will help.

(You can see the rendered docs at https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html.)

[Manishearth]

@jyn514 I think what we really need is a set of scenarios like:

• I want to modify and test the stdlib
• I want to modify and test rustdoc
  • and test its impact on the stdlib
• i want to modify and test the compiler
  • and test its impact on the stdlib

There's some of this in the first chapter, but there really should be more.

[Manishearth]

But the docs about what the stages themselves mean are good! That and x.py saying "building" vs "copying" has been helpful.

[jyn514]

I have a proposal for making this simpler in https://jyn.dev/2023/01/12/Bootstrapping-Rust-in-2023.html.

[jyn514]

btw here is an updated progress message from x.py:

; x b --dry-run --stage 1
Building bootstrap
    Finished dev [unoptimized] target(s) in 0.01s
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
Building compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
Building stage1 library artifacts (x86_64-unknown-linux-gnu)
Building stage1 tool rust-analyzer-proc-macro-srv (x86_64-unknown-linux-gnu)
Building tool rustdoc (stage0 -> stage1, x86_64-unknown-linux-gnu)
Build completed successfully in 0:00:00

hopefully stage0 -> stage1 makes more sense than just stage0.

[Manishearth]

I wonder if this direction may help further?

; x b --dry-run --stage 1
Building bootstrap
    Finished dev [unoptimized] target(s) in 0.01s
Building stage0 library artifacts (stage0 compiler -> stage0 library, x86_64-unknown-linux-gnu)
Building compiler artifacts (stage0 compiler -> stage1 compiler, x86_64-unknown-linux-gnu)
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
Building stage1 library artifacts (stage1 compiler -> stage1 library, x86_64-unknown-linux-gnu)
Building stage1 tool rust-analyzer-proc-macro-srv (stage1 compiler -> stage1 tool, x86_64-unknown-linux-gnu)
Building tool rustdoc (stage0 compiler -> stage1 tool, x86_64-unknown-linux-gnu)
Build completed successfully in 0:00:00

Interesting to me that rust-analyzer-proc-macro-srv is same-stage but most tools are cross-stage