Support for comments

[cmtt]

As follow-up for #99, this issue is about adding optional support for comment nodes.

As @maciejhirsz suggested, comments should be gathered by the lexer as internal list.
This way, f.e. all comments of a function can be pulled before the declaration.

Please note that there are three types of valid comments in JavaScript:

• Line comments // foo
• Block comments /* bar */
• HTML comments <!-- baz --> (:angry:)

[matklad]

If you want to support not only comments, but white space as well, and in general support loseless syntax trees, I suggest taking a look at Swift’s libsyntax architecture:
https://github.com/apple/swift/tree/master/lib/Syntax.

This might not be what you want though: such full fidelity syntax trees are heavier than ASTs.

[maciejhirsz]

Oh this is clever! I like the idea of trivia being attached to nodes, I wouldn't necessarily do it on every node (although, given how fast Ratel is, it might not be that much of a burden), but doing it on things like functions should be sufficient. It would require some re-parsing of comments, but then any tool that wants to read documentation or JSDoc types from comments is likely going to have to parse those in full anyway.

[matklad]

BTW, I have a crate with implenets a similar flavor of syntax trees here: https://github.com/rust-analyzer/rowan

It's main focus is loselessness, and not perf, so it has a ton of indirection and allocation.

[TitanNano]

I use ratel to do some JavaScript type analysis and I'm going to need comments, so I can parse JSDoc annotations. Is this something you are interested to implement in the near future?

[maciejhirsz]

@TitanNano my plan right now (which is just in my head) is for Logos to expose trivia as just a string of bytes between the previous and current token, and have that string attached to few nodes where comments might be useful (functions, methods, variable declarations), so you could extract JSDoc comments out of it.