[Go] Thread-safe ANTLR codegen

[TristonianJones]

The ANTLR Go codegen allocates mutable global variables which can result in data races as evidenced in google/cel-go#177 and reported in #2040. As a local workaround, global variables in the generated source were shifted into local variables within the NewLexer and NewParser functions.

While the change made the generated code thread-safe, it came at a significant CPU and memory overhead in parse-heavy call paths. However, exporting the SetInputStream method on the lexer.go makes it possible to use sync.Pool objects with the modified codegen artifacts and reduces the parsing overhead to within striking distance of the prior non-thread-safe code.

To take advantage of such thread-safety and object reuse within ANTLR Go applications, developers would need to regen their grammars and incorporate the following pattern of use within their own code base:

import (
  "sync"
  "github.com/antlr/antlr4/runtime/Go/antlr"
)

var(
  lexerPool *sync.Pool = &sync.Pool{ New: func() interface{} {
    return New<LexerName>(nil)
  }}
  parserPool *sync.Pool = &sync.Pool{ New: func() interface{} {
    return New<ParserName>(nil)
  }} 
)

func Parse(text string) antlr.ParseTree {
   lex := lexerPool.Get().(*<LexerName>)
   defer lexerPool.Put(lex)
   par := parserPool.Get().(*<ParserName>)
   defer parserPool.Put(par)

   lex.SetInputStream(antlr.NewInputStream(text))
   par.SetInputStream(antlr.NewCommonTokenStream(lex, 0))
   return par.<StartRule>()
}

All Go-related tests have been run and executed successfully following this change:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.antlr.v4.test.runtime.go.TestListeners
..Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.444 sec - in org.antlr.v4.test.runtime.go.TestListeners
Running org.antlr.v4.test.runtime.go.TestSemPredEvalLexer
.Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.974 sec - in org.antlr.v4.test.runtime.go.TestSemPredEvalLexer
Running org.antlr.v4.test.runtime.go.TestCompositeLexers
.Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.694 sec - in org.antlr.v4.test.runtime.go.TestCompositeLexers
Running org.antlr.v4.test.runtime.go.TestLexerErrors
..Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.991 sec - in org.antlr.v4.test.runtime.go.TestLexerErrors
Running org.antlr.v4.test.runtime.go.TestLexerExec
.....Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.283 sec - in org.antlr.v4.test.runtime.go.TestLexerExec
Running org.antlr.v4.test.runtime.go.TestParserErrors
.....Tests run: 34, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 26.256 sec - in org.antlr.v4.test.runtime.go.TestParserErrors
Running org.antlr.v4.test.runtime.go.TestParseTrees
.Tests run: 10, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.979 sec - in org.antlr.v4.test.runtime.go.TestParseTrees
Running org.antlr.v4.test.runtime.go.TestParserExec
......Tests run: 37, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 29.849 sec - in org.antlr.v4.test.runtime.go.TestParserExec
Running org.antlr.v4.test.runtime.go.TestSets
.....Tests run: 32, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.153 sec - in org.antlr.v4.test.runtime.go.TestSets
Running org.antlr.v4.test.runtime.go.TestFullContextParsing
..Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.256 sec - in org.antlr.v4.test.runtime.go.TestFullContextParsing
Running org.antlr.v4.test.runtime.go.TestPerformance
.Tests run: 7, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.369 sec - in org.antlr.v4.test.runtime.go.TestPerformance
Running org.antlr.v4.test.runtime.go.TestCompositeParsers
..Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.836 sec - in org.antlr.v4.test.runtime.go.TestCompositeParsers
Running org.antlr.v4.test.runtime.go.TestLeftRecursion
................Tests run: 98, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.905 sec - in org.antlr.v4.test.runtime.go.TestLeftRecursion
Running org.antlr.v4.test.runtime.go.TestSemPredEvalParser
....Tests run: 26, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 19.584 sec - in org.antlr.v4.test.runtime.go.TestSemPredEvalParser

Results :

Tests run: 340, Failures: 0, Errors: 0, Skipped: 8

[geektcp]

LGTm

[TristonianJones]

@pboyer would you be able to review and approve this change? Thanks!

[pboyer]

Thanks @TristonianJones! I approve of this change!

Like you said, this will cause significant performance regressions in clients that naively upgrade. We should mention this in the release notes. IMHO, the benefits clearly outweigh the drawbacks.

[TristonianJones]

Excellent @pboyer! Thanks for the review. I've been using ANTLR since v2, in one capacity or another, and I'm happy to be contributing back.

I agree this is release notes worthy. I debated adding a generated Parse method that would return an antlr.ParseTree and use sync.Pool instances under the covers, but it implied too much about how the parser would be used to be appropriate as part of the generated API. However, such instructions might be good to include in release notes.

What are the next steps here? Should I add @parrt in order to merge the changes?

[parrt]

@pboyer I think you are in charge of this target, right? Should I merge?

[pboyer]

Yup, merge.