Rustdoc needs to handle conditional compilation somehow

[brson]

We have some functions that are reimplemented per architecture, with docs only on one implementation. Rustdoc should ideally be able to merge these docs.

More seriously though a bunch of our libc hierarchy can't be viewed because it's conditionally compiled and the docs are built on linux.

Probably rustdoc should extract its documentation before pruning the AST for conditional compilation, collapse things with the same name into a single document, then run resolve.

[graydon]

I think the latter case is probably best, yes. Collapse-same-name is a bit of a kludge but not entirely unexpected.

[catamorphism]

Not backwards-compatible. Nominating for milestone 4, well-covered (having documentation tools work the way we expect is helpful in making sure the documentation covers everything...)

[graydon]

doc-coverage can be assured elsewhere, but this is embarrassing. should be fixed before production ready.

[graydon]

accepted for production-ready milestone

[msullivan]

Nothing new to say except to link to #5413.

[emberian]

#8125

This is not yet fixed by rustdoc_ng, and requires a bit of thought.

[emberian]

This is a hard problem with no good solution. If I use the AST before configuration, I can't have any hyperlinking for those items. Is that worth it?

[catamorphism]

Low, no milestone

[emberian]

My current thinking:

1. Parse, get unexpanded, unconfigured AST
2. Pull out everything cfg'd with the same name into sidetable
3. Configure, expand.
4. Pull out everything with the same name again.
5. Run rest of core.

Then, later, when cleaning, check if the item's nodeid is in the sidetable of cfg'd things. If so, add them to a vec of "alternatives" in the item, cleaning each.

This depends on:

1. The AST never being renumbered. (I don't think this is true today? @pcwalton?)
2. Holding onto a reference to the pre-expanded AST will cause that node to still be alive later on, but not otherwise affect the AST. (I think this is true today.)

When rendered, we'd have a list of "alternative definitions", listing the cfg's that would have enabled them. This of course doesn't handle things that are actually conditionally enabled/disabled, just things that are defined multiple times.

[pcwalton]

I'm not sure about AST renumberings off the top of my head.

[KostaCitrix]

Please fix. It's really confusing to see missing/wrong documentation if working on non-linux. This is one more 'gotcha' that you need to know, so this makes it harder to start working with rust on non-linux platform.

Thank you for you consideration! :)

[emberian]

@KostaCitrix this is a challenging issue to fix and we're all aware of the usability impacts this has. Demanding "please fix" doesn't really help solve the technical issues.

[tomjakubowski]

This is potentially less of a problem now that docs are installed with the distribution. It does seem, though, like this fact isn't particularly well known to newcomers.

[steveklabnik]

In some ways, but it still doesn't lay out a good welcome mat to those who aren't using Linux.

[retep998]

I ran out of fingers to count the number of times I've seen people not find Windows specific extension traits because the online docs were linux-only.

[tomjakubowski]

How about building docs on all three supported platforms and uploading to doc.rust-lang.org/nightly/windows, nightly/linux, etc?

Of course this doesn't solve the general problem of rustdoc and conditional compilation.

On Jul 16, 2015, at 19:32, Peter Atashian notifications@github.com wrote:

I ran out of fingers to count the number of times I've seen people not find Windows specific extension traits because the online docs were linux-only.

—
Reply to this email directly or view it on GitHub.

[lilyball]

One potential problem with trying to do the AST approach is if the same function/type is defined for multiple platforms with different doc comments, which doc comment do we show for the item? Or maybe even more importantly, same function with different types, or same type with different public fields?

One possible solution might be to add a selector somewhere on the page that lets you pick the platform you wish to view, and it would use that to show the correct documentation/types/fields. It could still show AST elements that aren't available on that platform but perhaps greyed out, to indicate that they're not available.

Another solution is to do what @tomjakubowski suggested and just upload docs for each platform to a different path. The main rustdoc UI could then still have the selector, which changes the URL to view that platform's documentation (and sets a cookie that is used by the non-platform-specific documentation path to automatically jump to the platform-specific path). One downside to this is every documentation URL that people share would then be a platform-specific URL.

[jonas-schievink]

One potential problem with trying to do the AST approach is if the same function/type is defined for multiple platforms with different doc comments, which doc comment do we show for the item?

In this case, I think we should emit a warning (that can be made into an error via a command line flag, to make this useful for CI or local testing, but still preserves compatibility) and just pick one (maybe the native one for compatibility). If only one of these item has docs, this is a non-issue (which should be the preferred way to document this IMO).

Or maybe even more importantly, same function with different types, or same type with different public fields?

I'm not sure about what a type means to Rustdoc, but I don't think it uses type information. This would mean that at least type aliases are handled automatically: The type can be something like libc::size_t, which is defined to something different depending on platform, but still acts the same way.

Now all remaining cases should be rare, so we could emit a warning for them as well. Which item we actually pick... no idea, maybe the same item we would pick today (like above).

I could also imagine solving these 2 issues by just adding docs for both items and marking them with their cfg attribute (something that would be nice in any case).

How about building docs on all three supported platforms

http://doc.rust-lang.org/libc is apparently doing that already, but it's not the best from a usability perspective, and I don't think std is doing the same, which is the whole reason I wrote this ;) (needed some Windows-specific trait, didn't find it, got annoyed at Rustdoc)

[aochagavia]

Note that this also has an effect on the save-analysis API (and it will be an issue when the RLS is implemented). For instance: if you are working on Windows, but are editing Linux-only code, the RLS won't do anything for you (no code completion, no jump to definition, etc) since the edited code is effectively invisible to save-analysis.

cc @nrc

[steveklabnik]

crates.fyi is another solution to this issue /cc @onur

[nrc]

I don't think there is a good answer to this in rustdoc itself. The better answer (which Steve hints at) is that we should build and host docs for every platform, not just Linux.

[retep998]

Tell me when winapi's docs are finally up on crates.fyi

[steveklabnik]

Triage: this is still the holy grail. cfg(rustdoc) lets us do some things that this would be useful for, but not all of them. I'm not sure that fixing this is possible.

[SOF3]

With the any and all cfg values, is it inevitable that a fix of this issue leads to an exponential complexity of compilation by features? Macro processing can also use cfg! and conditionally invoke proc macros from another crate to generate an unpredictable combination of symbols, so it seems that rustdoc might need to compile the macro expansion for m*2^n times (m number of architecture targets, n number of features, and probably more cfg variables not mentioned). Might this become an exploit or otherwise an unintended high cost of doc compilation?

[eggyal]

@jonas-schievink in #1998 (comment):

One potential problem with trying to do the AST approach is if the same function/type is defined for multiple platforms with different doc comments, which doc comment do we show for the item?

In this case, I think we should emit a warning (that can be made into an error via a command line flag, to make this useful for CI or local testing, but still preserves compatibility) and just pick one (maybe the native one for compatibility).

It's not just doc comments. Definitions can vary (significantly) too.

And what about modules? For example, suppose we have:

#[cfg(foo)]
pub mod host {
    pub struct Foo(pub usize);

    pub struct Bar;
}

#[cfg(not(foo))]
pub mod host {
    pub struct Foo(pub i8);

    pub struct Qux;
}

fn foo() -> host::Foo { host::Foo(0) }

Are you saying we should arbitrarily choose one mod host for which we generate documentation, and ignore the other—in which case either host::Bar or host::Qux will be entirely omitted (and we've not made much progress)? Or do we merge these different module definitions to create a franken mod host in which we have both host::Bar and host::Qux but arbitrarily choose just one definition of Foo?

Or do we generate separate documentation for both modules, in which case to what will the documentation for the (non-conditionally compiled) function foo() link for its return type? Making an arbitrary choice here is, I think, even more confusing than the status quo.

I think the only sensible options are as set out by @lilyball in #1998 (comment):

One possible solution might be to add a selector somewhere on the page that lets you pick the platform you wish to view, and it would use that to show the correct documentation/types/fields. It could still show AST elements that aren't available on that platform but perhaps greyed out, to indicate that they're not available.

Another solution is to do what @tomjakubowski suggested and just upload docs for each platform to a different path. The main rustdoc UI could then still have the selector, which changes the URL to view that platform's documentation (and sets a cookie that is used by the non-platform-specific documentation path to automatically jump to the platform-specific path). One downside to this is every documentation URL that people share would then be a platform-specific URL.

[jonas-schievink]

I'm not really sure what I was thinking 4.5 years ago :)

[eggyal]

Perhaps an intermediate solution would be to separately compile docs for each (crate-configurable, tier 1s by default) target platform, then perform a diff/merge on the results? Indeed, rather than performing the diff/merge operation on HTML it could be time to add a more machine-readable output format to rustdoc and perform the operation on that? JSON perhaps?

Edit I see rustdoc had JSON output historically, but it was removed: #32698 contains some discussion around it.

[jonas-schievink]

Reading my comment from back then again, it seems fairly clear that it doesn't apply to the example given above:

One potential problem with trying to do the AST approach is if the same function/type is defined for multiple platforms with different doc comments, which doc comment do we show for the item?

In this case, I think we should emit a warning (that can be made into an error via a command line flag, to make this useful for CI or local testing, but still preserves compatibility) and just pick one (maybe the native one for compatibility).

It only concerns different versions of an item that both have different documentation, but @eggyal's example does not contain any documentation. This doesn't solve the issue with different item definitions depending on #[cfg] flags, of course.

[CAD97]

(checking oldest issues for fun and profit)

Should this be closed in favor of tracking in #43781?

[jyn514]

@CAD97 No. #43781 is one possible solution to this issue, which has not be stabilized and has design concerns (https://rust-lang.zulipchat.com/#narrow/stream/266220-rustdoc/topic/Discussion.20about.20.60doc_cfg*.60.20features.20stabilization/near/304190110). This tracks the general issue of conditional compilation, not a specific feature.