Extended linking kinds

[retep998]

This RFC proposes adding a kind for object files which are passed directly to the linker.

Please bikeshed vigorously on the name of the kind.

rendered

[retep998]

cc @alexcrichton

[nikomatsakis]

I am very sympathetic to the aims of this RFC -- and I think I am persuaded that there is something missing in the current model, though I have to try to bring the details back into cache -- but I'd appreciate it you could edit it to maintain a more objective tone. I find phrases like "no way to properly link" or describing things as "weird" distracting, since they imply value judgements, which complicates the issue.

I'd rather see something like:

"The current set of linker options do not allow static linking to be handled by the linker. The current link=static causes rustc to bundle up the executables and kind=dynamic uses dynamic linking. However, on windows, a common configuration is to do blah blah blah."

[nikomatsakis]

One other thing: this RFC is a golden opportunity to lay out the subtleties involved here (and there are many, it seems). It'd be really valuable to have a good exposition of the windows linking model and where Rust is falling short.

[retep998]

I'm considering extending this RFC to also add kind=resource for foo.res resource files and kind=object for foo.obj object files. Would anyone be interested in these additions? This would also provide a more convenient way to deal with rust-lang/rust#33004

cc @DanielKeep @cybergeek94

[DanielKeep]

@retep998 At that point, wouldn't kind=dont_look_dont_touch_just_pass be better? Would kind=resource and kind=object do anything other than just passing the files along? Also, whilst object should map to other linkers, what should rustc do when it's told to link a resource, and the backend is ld? (I'm asking because I really have no idea.)

All that having been said: being able to link in resources and object files without having to use the gcc crate to build a fake static library first would be nice.

[retep998]

I'd kinda prefer kind=object and kind=resource since they are explicit about your intent, and they also don't allow completely arbitrary linker arguments to be passed. It's already kinda weird in that you can do rustc main.rs -l dylib=/out:foo and end up with a binary called foo.lib.

[retep998]

Just a bit of clarification on how resource files get passed in.

The main difference between using the MS resource tools and the GNU tools is that MS rc generates a ".res" file in a special binary resource format, which can be passed directly to MS link, while the GNU linker ld only supports resources in ".o" (same as ".obj") format. source

So this means for msvc you'd use kind=resource with a .res file you got a from a .rc file using rc.exe. For gnu you'd use windres.exe to create a .o from your .rc and pass it with kind=object. Does this seem fine?

[nrc]

the kind=object seems more elegant to me. Rather than get into the details of how rustc interacts with the linker, it is about operating in a mode where rustc does the absolute minimum and the user is then responsible for linking. That seems like a good scenario to be able to handle.

I am less convinced by the utility of kind=resource, aiui, kind=object would have the same effect. Although I see the point about intent, it seems a bit sub-optimal to add such a facility which is targeted at a single linker on a single platform.

[retep998]

@nrc We could make it so that with both gnu and msvc you'd pass your .rc file using kind=resource and then rustc does all the hard work of shelling out to rc.exe or windres.exe and passing it along appropriately to the linker.

[brson]

This sentence seems to be the key motivation, but I don't understand it: "kind=static doesn't work because it causes rustc to bundle the library into the *.rlib instead of passing it along to the linker, which results in rustc not looking in the standard library paths for the library."

What is the difference between "passing it along to the linker" and not doing that? Surely the code is linked either way right?

[retep998]

@brson The linker has a set of paths that it looks in for libraries. rustc does not know what those paths are. Suppose I have a static library located in one of those folders that the linker normally looks in for libraries. If I use kind=static, rustc will look for the library instead of the linker. Since rustc doesn't know about the path where the static library is, it will fail to locate it. Meanwhile using kind=betterstatic, rustc simply tells the linker to link that library, and at link time the linker looks through its paths and does find the library, and things end up working.

[brson]

Thanks for the explanation.

As an alternative proposal, what if we taught rustc about the pre-defined linker paths? The separation between the linker and compiler in rustc is artificial, a historical accident. If it worked the way we wished the linker would be integrated into rustc and rustc would know whatever the linker knows.

With the actual state of the world where linkers are ancient monoliths, adding this knowledge to rustc would not be simple, but there's probably a way.

[retep998]

When bundling another library into a Rust .rlib instead of passing it to the linker, there are other issues as well. For example if that library was compiled with LTCG, passing it to the linker will work fine, but if rustc bundles it into the .rlib the LTCG data is ruined. rust-lang/rust#26003

Actually, how about we try coming up with arguments for when kind=static would be better than kind=betterstatic.

1. Since the library is bundled into the .rlib, you don't have to keep the library around for link time, so that is one less thing to pass to the linker. Although this isn't useful in standard Cargo scenarios since the library will always be around for link time anyway and you're not manually invoking the linker.

Can anyone think of any other reasons?

[eddyb]

I much prefer this alternative. We generally make no guarantees about independent uses of anything other than executables, cdylibs and staticlibs, the last of which requires the user to make note of the library list Cargo outputs.

[retep998]

I am in favor of this alternative as well.

[retep998]

I updated the RFC to add kind=object. I didn't add kind=resource as it is fairly simple to use rc.exe and cvtres.exe for msvc or windres for mingw to create object files that can be passed along to the linker using kind=object.

[retep998]

Note that for resource files, the user cannot simply use lib.exe to create a .lib from the compiled .obj. A resource file compiled into a .obj and passed to the linker will work correctly, while if it is compiled into a .lib, it will no longer work. Thus there is a very real and legitimate need for kind=object.

[aturon]

Ping @rust-lang/tools, this RFC seems to've stalled. Can someone summarize the current state of play and try to make progress?

[retep998]

static-nobundle got implemented via the dllimport RFC, so at the moment this RFC is just proposing to add linking to object files.

[alexcrichton]

Ok cool if we don't want support for arbitrary object files then focusing on just resource files sounds good to me. I wonder though if could perhaps go for a bit more of an ergonomic inclusion, as well as lifting some requirements? I could imagine, for example:

• Could we find resource files similarly to how we find source code? That is if you say foo.rc it looks in the current directory of the current module for foo.rc? Or are foo.rc files generated and not handwritten?
• First off, is there a use case for allowing resource files in libraries? If not then we could just make that a hard error and not worry about saving paths or such.
• If we do want to allow resource files in libraries, perhaps we could support them explicitly without having to save off paths? That is, we could just put the rc file into the rlib and then whenever we link the rlib we take it out and pass it manually to the linker.

[retep998]

Could we find resource files similarly to how we find source code? That is if you say foo.rc it looks in the current directory of the current module for foo.rc? Or are foo.rc files generated and not handwritten?

Allowing resource files to be generated by a build script would be desirable, so anything that prevents that from working probably wouldn't be a good idea.

First off, is there a use case for allowing resource files in libraries? If not then we could just make that a hard error and not worry about saving paths or such.

A library may wish to bring along a resource file to provide some things that it needs. For example a UI library may want to define some resources so that it can use them. So resource files from a library definitely seems like a worthwhile use case to support.

If we do want to allow resource files in libraries, perhaps we could support them explicitly without having to save off paths? That is, we could just put the rc file into the rlib and then whenever we link the rlib we take it out and pass it manually to the linker.

We don't necessarily have to compile the resource file immediately, saving it somewhere and later compiling it right before passing it to the linker is fine. However what we cannot do is put the compiled resource file in a library and pass that library to the linker, because the linker will fail to recognize the compiled resource file in the library.

[alexcrichton]

Ok so it seems to be like we may not want to use #[link] for this if we're just slurping up resource files (either handwritten or generated) because that's related moreso to extern blocks and linking native libraries.

Perhaps something like:

#[windows_resource = "foo.rc"]
extern {} // ??

That way if you want to include a file relative to a source file you can do so. If you want to generate it via a build script then you can do so and include that via include!

I kinda like the idea of not running rc and/or windres until the very end. That way we just put the files in rlibs and forget about them. At the end we extract all resource files, compile them all, and pass them all to the linker.

I'm not really sure where or how to precisely specify the inclusion of a resource file, though. We could maybe slap it on a module or something like that, but it seems like almost a hack no matter how we slice it unfortunately.

[retep998]

@alexcrichton Oh, there might actually be an issue with storing resource files to compile them later, because when resource files are compiled they pull in other files around them, like icons and manifests and stuff. If you store the rc then those other files won't be around when you need them. So you'll have to compile the resource file on the spot and then somehow keep track of or store the compiled .res file to later pass to the linker.

[alexcrichton]

What are your thoughts on how resource files are declared? I feel like #[link] is the wrong place to describe exact paths to files (as that's not elsewhere what it's used for today) and a custom attribute here is warranted, but I'm curious what you think.

[retep998]

I personally don't care too much about what the syntax for declaring resource files is. If someone proposes a better syntax that others agree on, then I'll gladly update the RFC.

[alexcrichton]

Ok well in that case my proposal is:

#[windows_resource = "foo.rc"]
extern {}

The foo.rc file here is located with the same logic as #[path] (e.g. in the same dir by default) and the compiler also verifies that the extern block is empty. I feel that this is more suitable than #[link] because it's using the existing infrastructure we have today to find source files (which I'm currently under the impression an rc file is), and it doesn't reuse #[link] which would have very different lookup semantics.

This also differentiates itself as a windows-specific extension (like #![windows_subsystem]) as opposed to throwing it into the ring with #[link]

[Boscop]

But why does it need an extern block? Why not as a crate attribute?

[retep998]

@alexcrichton What would be the recommended way to link a resource file that is generated using a build script using your attribute?

[vadimcn]

I kinda agree with @retep998: linking in an object file (not necessarily resources) is just something you sometimes need to do when building for a custom target.
Right now you basically have to dump linker args with -Zprint-link-args and then run it yourself, which soon will only be possible on nightly.

[alexcrichton]

@Boscop with a crate attribute it's difficult to include something dynamically generated, such as (to answer @retep998's question as well) with an extern block you can generate $OUT_DIR/foo.rs with the extern contents and then in your code include!(concat!(env!("OUT_DIR"), "/foo.rs"));

@vadimcn just to clarify, but you're pushing back on the trimming down of this RFC from object files to just resource files? If so, can you clarify why archives wouldn't work? (e.g. why raw object files must be used)

[retep998]

Having to generate a rust file in order to link to a generated resource file seems rather convoluted. I'd rather have a rustc flag to link in a resource file and cargo would support some sort of cargo:windows_resource output from build scripts to specify that flag.

[vadimcn]

@alexcrichton: I liked the part about linking object files in general, not just outputs from windres/cvtres. On more than one occasion during my hacking I had wished I could pass objects directly to Rust compiler, like I can do with GCC. It would also let us deal with resources, so we'd be killing two birds with one stone :)

[michaelwoerister]

Agreed, being able to just link to object files would be nice. Maybe rustc could look through -L paths for object files. Since those won't come from system directories, it might be OK not to rely on the linker for finding those.

[alexcrichton]

I would also be ok with linking arbitrary object files with logic that looks like:

• The compiler finds an object file (by path) via searching
• If building an rlib, the compiler puts the object in the rlib
• If building an executable, just pass the object
• If building an executable with an object in an rlib, take the object out, modify the rlib, and then pass those all to the linker

I don't feel that it's strictly necessary, but I'm comfortable with those semantics and if we want to go back to arbitrary object files that'd be ok.

[retep998]

Since those won't come from system directories, it might be OK not to rely on the linker for finding those.

Unfortunately, this is wrong! Due to the overriding nature of object files in the msvc toolchain, msvc actually provides a set of object files to override certain behavior. If you want to link to one of those object files, the linker must be able to link to those object files, and expecting the rlib to adequately inform rustc of where those object files is just not going to work.

If building an rlib, the compiler puts the object in the rlib

Due to the above, this step won't work, unless there are major changes to how rustc searches for libraries.

[alexcrichton]

Discussed at the @rust-lang/tools triage a few days ago, the conclusion we reached were:

• It seems prudent to refocus this RFC on object files instead of just resource files. In addition to what's been brought up so far we'd have to otherwise expose configuring the resource file compiler path and also arguments to the resource file compiler, both of which are just more easily done if we operate through object files.
• We should use -L for lookup paths to object files, and the compiler itself should be responsible for finding these files. If an extension is missing the compiler can look for foo.o or foo.obj, but if an extension is already specified the compiler will look for that file instead.

Does that make sense? How does that sound to you @retep998?

[retep998]

We should use -L for lookup paths to object files, and the compiler itself should be responsible for finding these files.

So what happens if I want to link a system object file, like one of the object files provided by VC++?

[codyps]

IMO, the single namespace for native libs (due to using lookup paths) is already a problem. Having to deal with a single namespace for objects (again, due to the proposed use of lookup paths) doesn't seem like a good idea if it's not a problem to specify full paths.

[alexcrichton]

@retep998 I'm not sure, you tell me what would happen!

@jmesmon I think it'd be fine to accept full paths on the command line, but objects may also want to be encoded into the source file itself in which case a full path may not always be desired.

[retep998]

@alexcrichton Well, if rustc were responsible for finding the object files and not the linker, I'd be screwed. This is the same thing that happened with kind=static, where I couldn't link to static system libraries. It's simply way too complicated for the user to find where those system object files are when the linker already knows where to look (because either an environment variable or rustc told it).

[alexcrichton]

Ok, do you have a proposal to fix that problem?

[retep998]

The problem is that link.exe is the only linker that supports searching library paths for arbitrary object files. ld will search for libfoo if you specify -lfoo so that won't do to find foo.o while just passing foo.o to ld won't search the library paths. Here are a couple options I thought of:

1. gnu and msvc have somewhat different behavior. While they will both attempt to pass absolute paths when possible and for relative paths they will try to locate the object file to pass an absolute path, gnu will simply error if rustc cannot find the object file, while msvc will trust that the object file can be found in one of the library paths and leaves it up to the linker to actually error.
2. msvc has its own special kind specifically for system object files. If other linkers end up supporting this, we can extend this kind to those linkers.

[alexcrichton]

Could you elaborate on the motivation for system object files? Could that perhaps get punted to a future RFC?

Could you also elaborate what your first option would look like? E.g. the specifics of lookup.

[aturon]

I'm going to close this RFC for the time being, to keep our queue clean, as it's waiting on follow up from its author, @retep998. Please feel free to reopen if/when you're ready to pick this topic back up.