New unreachable token value warning #2066

KvanTTT · 2017-10-22T13:14:46Z

Partial fix for #1072 (see unit tests).

I tested my version on grammars-v4 repository and found a couple of such warnings:

warning(184): LessLexer.g4:224:0: One of the token FORMAT values unreachable. % is always overlapped by token PERC
warning(184): ZLexer.g4:478:0: One of the token ID0 values unreachable. _ is always overlapped by token ARGUMENT

Also I discovered and fixed such warnings in ANTLR unit tests itself (testInvalidLexerCommand, testLexerCommandArgumentValidation).

parrt · 2017-10-27T15:30:47Z

tool/src/org/antlr/v4/semantics/SymbolChecks.java

@@ -96,8 +89,7 @@ public void checkActionRedefinitions(List<GrammarAST> actions) {
 			nameNode = (GrammarAST) ampersandAST.getChild(0);
 			if (ampersandAST.getChildCount() == 2) {
 				name = nameNode.getText();
-			}
-			else {
+			} else {


hi. please use same style. } on line by themselves

parrt · 2017-10-27T15:31:07Z

tool/src/org/antlr/v4/semantics/SymbolChecks.java

-     *  for repeated defs.
-     */
-    public void checkForLabelConflicts(Collection<Rule> rules) {
+	/**


are you changing whitespace by mistake?

Сonversely I replaced all whitespaces on tabs (respect to general project style).

parrt · 2017-10-27T15:32:52Z

tool/src/org/antlr/v4/semantics/SymbolChecks.java

@@ -341,42 +318,130 @@ public void checkForModeConflicts(Grammar g) {
 		}
 	}

+	public void checkForUnreachableTokens(Grammar g) {


can you explain in a comment your strategy for finding unreachable tokens? I also need to review that strategy to know if it is correct. :)

KvanTTT · 2017-10-28T20:20:29Z

Please wait, do not merge. I want to implement yet another check.

KvanTTT · 2017-11-01T20:21:19Z

Probably the complexity can be reduced down to linear. I'll take a look until the end of the week.

sharwell

Need to remove the unrelated formatting changes from this.

sharwell · 2017-11-01T20:24:11Z

tool/src/org/antlr/v4/semantics/SymbolChecks.java

@@ -223,11 +207,11 @@ private void checkForTypeMismatch(Rule r, LabelElementPair prevLabelPair, LabelE
 					labelPair.type + "!=" + prevLabelPair.type);
 		}
 		if (!prevLabelPair.element.getText().equals(labelPair.element.getText()) &&
-			(prevLabelPair.type.equals(LabelType.RULE_LABEL) || prevLabelPair.type.equals(LabelType.RULE_LIST_LABEL)) &&
-			(labelPair.type.equals(LabelType.RULE_LABEL) || labelPair.type.equals(LabelType.RULE_LIST_LABEL))) {
+				(prevLabelPair.type.equals(LabelType.RULE_LABEL) || prevLabelPair.type.equals(LabelType.RULE_LIST_LABEL)) &&


📝 Previous indentation was correct here:

antlr4/runtime/Java/nb-configuration.xml

Line 36 in 64ae36b

<org-netbeans-modules-editor-indent.text.x-java.CodeStyle.project.indent-shift-width>4</org-netbeans-modules-editor-indent.text.x-java.CodeStyle.project.indent-shift-width>

sharwell · 2017-11-01T20:26:48Z

tool/src/org/antlr/v4/semantics/SymbolChecks.java

 		}
 	}

 	protected void checkReservedNames(Collection<Rule> rules) {
 		for (Rule rule : rules) {
 			if (reservedNames.contains(rule.name)) {
-				errMgr.grammarError(ErrorType.RESERVED_RULE_NAME, g.fileName, ((GrammarAST)rule.ast.getChild(0)).getToken(), rule.name);
+				errMgr.grammarError(ErrorType.RESERVED_RULE_NAME, g.fileName, ((GrammarAST) rule.ast.getChild(0)).getToken(), rule.name);


📝 Previous spacing was correct here:

antlr4/runtime/Java/nb-configuration.xml

Line 43 in 64ae36b

<org-netbeans-modules-editor-indent.text.x-java.CodeStyle.project.spaceAfterTypeCast>false</org-netbeans-modules-editor-indent.text.x-java.CodeStyle.project.spaceAfterTypeCast>

sharwell · 2017-11-01T20:29:41Z

tool/src/org/antlr/v4/semantics/SymbolChecks.java

-			GrammarAST arg = (GrammarAST)ref.getFirstChildWithType(ANTLRParser.ARG_ACTION);
-			if ( arg!=null && (r==null || r.args==null) ) {
+			GrammarAST arg = (GrammarAST) ref.getFirstChildWithType(ANTLRParser.ARG_ACTION);
+			if (arg != null && (r == null || r.args == null)) {


📝 Spacing here was generally unspecified (@parrt uses the previous spacing, while I use the updated spacing). Generally accepted practice became not changing it unless you were rewriting the line.

sharwell · 2017-11-01T20:30:36Z

tool/src/org/antlr/v4/tool/ErrorType.java

@@ -522,15 +522,15 @@
 	 */
 	USE_OF_BAD_WORD(134, "symbol <arg> conflicts with generated code in target language or runtime", ErrorSeverity.ERROR),
 	/**
-	 * Compiler Error 134.
+	 * Compiler Error 183.


💭 Wasn't this part of another pull request?

Yes, it is, but already merged.

sharwell · 2017-11-01T20:32:30Z

tool-testsuite/test/org/antlr/v4/test/tool/TestSymbolIssues.java

+				"TOKEN7: 'qwer'+;\n" +
+				"TOKEN8: 'a' 'b' | 'b' | 'a' 'b';\n" +
+				"fragment\n" +
+				"TOKEN9: 'asdf' | 'qwer' | 'qwer';\n" +


💡 Should include tokens with characters that need escapes, to make sure the values are properly escaped in the error messages.

…rror type. tabs normalized.

…methods.

parrt · 2017-11-29T17:44:33Z

Ok, are we ready to go with this merge?

KvanTTT · 2017-11-29T17:54:15Z

@sharwell please check last fixes.

parrt · 2017-12-06T20:29:16Z

@KvanTTT How many grammars have you run this on?

KvanTTT · 2017-12-06T20:32:06Z

All grammars from an official grammars-v4 repository, see my first comment: #2066 (comment)

parrt reviewed Oct 27, 2017

View reviewed changes

sharwell suggested changes Nov 1, 2017

View reviewed changes

KvanTTT added 9 commits November 25, 2017 16:24

Added checkForUnreachableTokens method, added new TOKEN_UNREACHABLE e…

da57241

…rror type. tabs normalized.

Added testUnreachableTokens unit-test.

585325f

Fixed TOKEN_UNREACHABLE warning checks.

328cdf3

Extend and fix unit tests.

9324d07

} on line by themselves.

58349e5

Added comments to checkForUnreachableTokens and getSingleTokenValues …

f4c4e8b

…methods.

Added check for fragment rules.

64b2ced

Restored formatting.

5b11aed

Added tests with escaped characters.

763eccc

KvanTTT force-pushed the unreachable-tokens branch from 9ebf9e6 to 763eccc Compare November 25, 2017 13:24

Yes another formatting fixes.

07a0a80

parrt added this to the 4.7.1 milestone Nov 29, 2017

parrt added comp:tool error-handling labels Nov 29, 2017

This was referenced Dec 4, 2017

Add a new CharStream that converts the symbols to upper or lower case. #2046

Closed

Import lexers with modes into other lexer grammars #2135

Merged

Classic gotcha of mismatched input 'xyz' expecting 'xyz' #1072

Open

parrt merged commit 489a11b into antlr:master Dec 6, 2017

parrt added the type:improvement label Dec 8, 2017

KvanTTT deleted the unreachable-tokens branch November 8, 2021 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New unreachable token value warning #2066

New unreachable token value warning #2066

KvanTTT commented Oct 22, 2017

parrt Oct 27, 2017

parrt Oct 27, 2017

KvanTTT Oct 28, 2017

parrt Oct 27, 2017

KvanTTT Oct 28, 2017

KvanTTT commented Oct 28, 2017 •

edited

Loading

KvanTTT commented Nov 1, 2017

sharwell left a comment •

edited

Loading

sharwell Nov 1, 2017

sharwell Nov 1, 2017

sharwell Nov 1, 2017

sharwell Nov 1, 2017

KvanTTT Nov 25, 2017 •

edited

Loading

sharwell Nov 1, 2017

KvanTTT Nov 29, 2017

parrt commented Nov 29, 2017

KvanTTT commented Nov 29, 2017 •

edited

Loading

parrt commented Dec 6, 2017

KvanTTT commented Dec 6, 2017

New unreachable token value warning #2066

New unreachable token value warning #2066

Conversation

KvanTTT commented Oct 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KvanTTT commented Oct 28, 2017 • edited Loading

KvanTTT commented Nov 1, 2017

sharwell left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KvanTTT Nov 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parrt commented Nov 29, 2017

KvanTTT commented Nov 29, 2017 • edited Loading

parrt commented Dec 6, 2017

KvanTTT commented Dec 6, 2017

KvanTTT commented Oct 28, 2017 •

edited

Loading

sharwell left a comment •

edited

Loading

KvanTTT Nov 25, 2017 •

edited

Loading

KvanTTT commented Nov 29, 2017 •

edited

Loading