Allow paths as tokenVocab option #1311
Conversation
|
Thanks @sebkur A minimal fix to a real problem. |
| int lastChar = vocabName.charAt(len - 1); | ||
| if (len >= 2 && firstChar == '\'' && lastChar == '\'') { | ||
| vocabName = vocabName.substring(1, len-1); | ||
| } |
There was a problem hiding this comment.
Removing the quotes could be done with less code:
String vocabName = tokenVocabNode.getText().replaceAll("\\A'|'\\Z", "");
It could be worthwhile to add a helper to misc.Utils. The Maven plugin also needs to strip quotes.
There was a problem hiding this comment.
Not sure, what your regex does exactly, but it also strips the ' from 'test or test'. I think making sure that we're dealing with a properly quoted string makes sense, i.e. that there's a quote character at the beginning and the end of the input.
There was a problem hiding this comment.
But 'test or test' is invalid input and rejected by ANTLR. So we can expect the string either to be fully quoted or not quoted at all and the regex removes the single quotes as desired.
As reported on the mailing list in 2007 and also in the Gradle forums in the last years, it is currently impossible to define two different grammars in different packages using separate lexer and parser files and import the lexer files using
tokenVocab.Ultimately the problem is that the referenced
tokenVocabcannot be resolved when processing a grammar that does not reside in the default package. Using the-libmakes it possible to work around this problem for a single package, but this fails to work for multiple grammars in different packages because-libcan only be specified once.This patch allows the
tokenVocabto specify paths instead of only file names, i.e. it is then possible to writetokenVocab='com/test/html/HTMLLexer';The ANTLR parser allows quoted strings for the
tokenVocabalready, and resolving the files just works when specifying a path. However, processing the fileset can still fail because the dependency graph is not built correctly. (Because the dependency graph builder does not identify thatHTMLLexerand'com/test/html/HTMLLexer';are references to the same thing.To make the dependency graph builder work, I added some code that removes the quotes and extracts only the filename from the specified
tokenVocab.I have created a testing repository that demonstrates the new behavior of the
Toolfor two example grammars from the grammars-v4 repository, that have been moved to packages to fit the use case.