libtest fails to link with lld at main

[andrewpollack]

Building Rust toolchain using latest lld linker at hash 6443c0ee02c7785bc917fcf508f4cc7ded38487a leads to an error when compiling libtest at stage0:

Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling cc v1.0.76
   Compiling core v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/core)
   Compiling libc v0.2.138
   Compiling memchr v2.5.0
   Compiling std v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/std)
   Compiling compiler_builtins v0.1.85
   Compiling unwind v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/unwind)
   Compiling profiler_builtins v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/profiler_builtins)
   Compiling rustc-std-workspace-core v1.99.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/alloc)
   Compiling cfg-if v1.0.0
   Compiling adler v1.0.2
   Compiling rustc-demangle v0.1.21
   Compiling rustc-std-workspace-alloc v1.99.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/rustc-std-workspace-alloc)
   Compiling panic_abort v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/panic_abort)
   Compiling panic_unwind v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/panic_unwind)
   Compiling gimli v0.26.1
   Compiling miniz_oxide v0.5.3
   Compiling std_detect v0.1.5 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/stdarch/crates/std_detect)
   Compiling hashbrown v0.12.3
   Compiling object v0.29.0
   Compiling addr2line v0.17.0
   Compiling rustc-std-workspace-std v1.99.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/rustc-std-workspace-std)
   Compiling proc_macro v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/proc_macro)
   Compiling unicode-width v0.1.10
   Compiling getopts v0.2.21
   Compiling test v0.0.0 (/b/s/w/ir/x/w/fuchsia-third_party-rust/library/test)
error: linking with `/b/s/w/ir/x/w/cipd/bin/clang++` failed: exit status: 1
  |
  = note: "/b/s/w/ir/x/w/cipd/bin/clang++" "-Wl,--version-script=/b/s/w/ir/x/t/rustctgpUpK/list" "-m64" "/b/s/w/ir/x/t/rustctgpUpK/symbols.o" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/test-3bd20e94ef73cd68.test.d3d4ef93-cgu.0.rcgu.o" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/test-3bd20e94ef73cd68.1w1vzuec12odskdf.rcgu.rmeta" "-Wl,--as-needed" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/release/deps" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/compiler_builtins-1f459caf2b14046b/out" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/build/profiler_builtins-a255468473e62d4f/out" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libgetopts-023b5f868ec7a238.rlib" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libunicode_width-e0099b6d5dcba458.rlib" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/librustc_std_workspace_std-8f7f1564c2f2e122.rlib" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps" "-Wl,-Bdynamic" "-lstd-77833ca844ade20f" "-Wl,-Bstatic" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcompiler_builtins-63346244df871953.rlib" "-Wl,-Bdynamic" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/b/s/w/ir/x/w/staging/build/fuchsia-build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libtest-3bd20e94ef73cd68.so" "-shared" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs" "--target=x86_64-unknown-linux-gnu" "--sysroot=/b/s/w/ir/x/w/cipd/linux" "-fuse-ld=lld" "-Wl,-z,origin" "-Wl,-rpath,$ORIGIN/../lib"
  = note: ld.lld: error: version script assignment of 'global' to symbol '__rust_alloc' failed: symbol not defined
          ld.lld: error: version script assignment of 'global' to symbol '__rust_alloc_error_handler_should_panic' failed: symbol not defined
          ld.lld: error: version script assignment of 'global' to symbol '__rust_alloc_zeroed' failed: symbol not defined
          ld.lld: error: version script assignment of 'global' to symbol '__rust_dealloc' failed: symbol not defined
          ld.lld: error: version script assignment of 'global' to symbol '__rust_realloc' failed: symbol not defined
          clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
          
error: could not compile `test` due to previous error

Error is coming from https://reviews.llvm.org/D135402 (Enable --no-undefined-version by default.), with the description indicating that our version scripts were not correct.

Local workaround is to pass -Clink-arg=-Wl,--undefined-version in Rustflags.

[chbaker0]

I ran into this too.

I wonder why rustc is attempting to export those symbols from test, but not std? I'm seeing that e.g. x.py build --stage 0 library/std works but x.py build --stage 0 library/test does not. I can't determine what's different between std and test that would cause only the latter to output an invalid version script.

[chbaker0]

This actually repros outside of test. If you attempt to compile any crate as a dylib, and std is available as a dylib and linked as such (e.g. with the -Cprefer-dynamic) flag, you can reproduce the error by simply passing -Clink-arg=-Wl,--no-undefined-version. This was with stable rustc and my system gcc.

Hopefully reproducing it outside of Rust's bootstrap will make it easier to debug.

[chbaker0]

related discussion: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/.60std.60.20dylib.20exports.20.60__rust_alloc.60/near/327158764

[chbaker0]

I suspect I found the issue:

First, early in compilation rustc determines if a crate needs an allocator (whether or not it comes from the crate itself, or a dep). This is filled in a crate metadata struct field:

rust/compiler/rustc_metadata/src/creader.rs

Lines 39 to 41 in 0b439b1

	/// This crate needs an allocator and either provides it itself, or finds it in a dependency.
	/// If the above is true, then this field denotes the kind of the found allocator.
	allocator_kind: Option<AllocatorKind>,

If Some(_), the field says whether the crate graph contains a #[global_allocator] or not, but that's not directly relevant to the bug. This field will usually be Some(_); it's None if std and alloc are not used.

Later, rustc checks whether to codegen the __rust_alloc et al functions:

rust/compiler/rustc_codegen_ssa/src/base.rs

Lines 618 to 649 in 0b439b1

	// Codegen an allocator shim, if necessary.
	//
	// If the crate doesn't have an `allocator_kind` set then there's definitely
	// no shim to generate. Otherwise we also check our dependency graph for all
	// our output crate types. If anything there looks like its a `Dynamic`
	// linkage, then it's already got an allocator shim and we'll be using that
	// one instead. If nothing exists then it's our job to generate the
	// allocator!
	let any_dynamic_crate = tcx.dependency_formats(()).iter().any(|(_, list)| {
	use rustc_middle::middle::dependency_format::Linkage;
	list.iter().any(|&linkage| linkage == Linkage::Dynamic)
	});
	let allocator_module = if any_dynamic_crate {
	None
	} else if let Some(kind) = tcx.allocator_kind(()) {
	let llmod_id =
	cgu_name_builder.build_cgu_name(LOCAL_CRATE, &["crate"], Some("allocator")).to_string();
	let module_llvm = tcx.sess.time("write_allocator_module", || {
	backend.codegen_allocator(
	tcx,
	&llmod_id,
	kind,
	// If allocator_kind is Some then alloc_error_handler_kind must
	// also be Some.
	tcx.alloc_error_handler_kind(()).unwrap(),
	)
	});
	Some(ModuleCodegen { name: llmod_id, module_llvm, kind: ModuleKind::Allocator })
	} else {
	None
	};

Notably, if any dependency is linked as a dylib, it will not generate them and instead link to the implementations exported from another dylib.

Independently, rustc determines which symbols to export when generating a dylib and provides this list to the linker. Part of this is deciding whether to export the allocator symbols:

rust/compiler/rustc_codegen_ssa/src/back/symbol_export.rs

Lines 200 to 226 in 0b439b1

	if tcx.allocator_kind(()).is_some() {
	for symbol_name in ALLOCATOR_METHODS
	.iter()
	.map(|method| format!("__rust_{}", method.name))
	.chain(["__rust_alloc_error_handler".to_string(), OomStrategy::SYMBOL.to_string()])
	{
	let exported_symbol = ExportedSymbol::NoDefId(SymbolName::new(tcx, &symbol_name));
	symbols.push((
	exported_symbol,
	SymbolExportInfo {
	level: SymbolExportLevel::Rust,
	kind: SymbolExportKind::Text,
	used: false,
	},
	));
	}
	symbols.push((
	ExportedSymbol::NoDefId(SymbolName::new(tcx, OomStrategy::SYMBOL)),
	SymbolExportInfo {
	level: SymbolExportLevel::Rust,
	kind: SymbolExportKind::Text,
	used: false,
	},
	));
	}

Notably it does not perform the check from the last step. Even if a dylib dependency is linked whose allocator symbols this crate should use, the export list still contains these symbol names. So when building a dylib that links another dylib, rustc tells the linker to export symbols that weren't actually defined. The library/test build failure is just one instance of this bug.

@bjorn3 who helped in the Zulip discussion. Can you confirm this explanation makes sense?

[bjorn3]

Most of it. There is a further complication that you can specify multiple crate types for a single compilation. It is then possible for some crate types to generate the allocator shim while others don't generate it. The list of exported symbols however is for all crate types at the same time.

One way to fix this would be to have a separate list of exported symbols for each crate type. Another option would be to move the addition of the __rust_* symbols to the actual code that invokes the linker.

[chbaker0]

Listing exported symbols per crate type seems like the most correct fix in principle, but the lists will be almost the same except for the allocator symbols. I'll try to find a clean way to do the latter.

[chbaker0]

@rustbot claim