Windows subsystem support

text/0000-windows-subsystem.md

@@ -0,0 +1,161 @@
+- Feature Name: Windows Subsystem
+- Start Date: 2016-07-03
+- RFC PR: ____
+- Rust Issue: ____
+
+# Summary
+[summary]: #summary
+
+Rust programs compiled for windows will always allocate a console window on
+startup. This behavior is controlled via the `SUBSYSTEM` parameter passed to the
+linker, and so *can* be overridden with specific compiler flags. However, doing
+so will bypass the rust-specific initialization code in `libstd`, as when using
+the MSVC toolchain, the entry point must be named `WinMain`.
+
+This RFC proposes supporting this case explicitly, allowing `libstd` to
+continue to be initialized correctly.
+
+# Motivation
+[motivation]: #motivation
+
+The `WINDOWS` subsystem is commonly used on windows: desktop applications
+typically do not want to flash up a console window on startup.
+
+Currently, using the `WINDOWS` subsystem from rust is undocumented, and the
+process is non-trivial when targeting the MSVC toolchain. There are a couple of
+approaches, each with their own downsides:
+
+## Define a WinMain symbol
+
+A new symbol `pub extern "system" WinMain(...)` with specific argument
+and return types must be declared, which will become the new entry point for
+the program.
+
+This is unsafe, and will skip the initialization code in `libstd`.
+
+The GNU toolchain will accept either entry point.
+
+## Override the entry point via linker options
+
+This uses the same method as will be described in this RFC. However, it will
+result in build scripts also being compiled for the windows subsystem, which
+can cause additional console windows to pop up during compilation, making the
+system unusable while a build is in progress.
+
+# Detailed design
+[design]: #detailed-design
+
+When an executable is linked while compiling for a windows target, it will be
+linked for a specific *subsystem*. The subsystem determines how the operating
+system will run the executable, and will affect the execution environment of
+the program.
+
+In practice, only two subsystems are very commonly used: `CONSOLE` and
+`WINDOWS`, and from a user's perspective, they determine whether a console will
+be automatically created when the program is started.
+
+## New crate attribute
+
+This RFC proposes two changes to solve this problem. The first is adding a
+top-level crate attribute to allow specifying which subsystem to use:
+
+`#![windows_subsystem = "windows"]`
+
+Initially, the set of possible values will be `{windows, console}`, but may be
+extended in future if desired.
+
+The use of this attribute in a non-executable crate will result in a compiler
+warning. If compiling for a non-windows target, the attribute will be silently
+ignored.
+
+## Additional linker argument
+
+For the GNU toolchain, this will be sufficient. However, for the MSVC toolchain,
+the linker will be expecting a `WinMain` symbol, which will not exist.
+
+There is some complexity to the way in which a different entry point is expected
+when using the windows subsystem. Firstly, the C-runtime library exports two
+symbols designed to be used as an entry point:
+```
+mainCRTStartup
+WinMainCRTStartup
+```
+
+`LINK.exe` will use the subsystem to determine which of these symbols to use
+as the default entry point if not overridden.
+
+Each one performs some unspecified initialization of the CRT, before calling out
+to a symbol defined within the program (`main` or `WinMain` respectively).
+
+The second part of the solution is to pass an additional linker option when
+targeting the MSVC toolchain:
+`/ENTRY:mainCRTStartup`
+
+This will override the entry point to always be `mainCRTStartup`. For
+console-subsystem programs this will have no effect, since it was already the
+default, but for windows-subsystem programs, it will eliminate the need for
+a `WinMain` symbol to be defined.
+
+This command line option will always be passed to the linker, regardless of the
+presence or absence of the `windows_subsystem` crate attribute, except when
+the user specifies their own entry point in the linker arguments. This will
+require `rustc` to perform some basic parsing of the linker options.
+
+# Drawbacks
+[drawbacks]: #drawbacks
+
+- A new platform-specific crate attribute.
+- The difficulty of manually calling the rust initialization code is potentially
+  a more general problem, and this only solves a specific (if common) case.
+- The subsystem must be specified earlier than is strictly required: when
+  compiling C/C++ code only the linker, not the compiler, needs to actually be
+  aware of the subsystem.
+- It is assumed that the initialization performed by the two CRT entry points
+  is identical. This seems to currently be the case, and is unlikely to change
+  as this technique appears to be used fairly widely.
+
+# Alternatives
+[alternatives]: #alternatives
+
+- Only emit one of either `WinMain` or `main` from `rustc` based on a new
+  command line option.
+
+  This command line option would only be applicable when compiling an
+  executable, and only for windows platforms. No other supported platforms
+  require a different entry point or additional linker arguments for programs
+  designed to run with a graphical user interface.
+
+  `rustc` will react to this command line option by changing the exported
+  name of the entry point to `WinMain`, and passing additional arguments to
+  the linker to configure the correct subsystem. A mismatch here would result
+  in linker errors.
+
+  A similar option would need to be added to `Cargo.toml` to make usage as
+  simple as possible.
+
+  There's some bike-shedding which can be done one the exact command line
+  interface, but one possible option is shown below.
+
+  Rustc usage:
+  `rustc foo.rs --crate-subsystem windows`
+
+  Cargo.toml
+  ```toml
+  [package]
+  # ...
+
+  [[bin]]
+  name = "foo"
+  path = "src/foo.rs"
+  subsystem = "windows"
+  ```
+
+  The `crate-subsystem` command line option would exist on all platforms,
+  but would be ignored when compiling for a non-windows target, so as to
+  support cross-compiling. If not compiling a binary crate, specifying the
+  option is an error regardless of the target.
+
+# Unresolved questions
+[unresolved]: #unresolved-questions
+
+None