When we hit an accept state in either the DFA or the ATN, we have to notify the character stream to start buffering characters via Antlr4.Runtime.IIntStream.Mark() and record the current state. The current sim state includes the current index into the input, the current line, and current character position in that line. Note that the Lexer is tracking the starting line and characterization of the token. These variables track the "state" of the simulator when it hits an accept state.

We track these variables separately for the DFA and ATN simulation because the DFA simulation often has to fail over to the ATN simulation. If the ATN simulation fails, we need the DFA to fall back to its previously accepted state, if any. If the ATN succeeds, then the ATN does the accept and the DFA simulator that invoked it can simply return the predicted token type.

LexerATNSimulator.SimState Class Documentation
コード例 #1
0
 protected internal virtual void CaptureSimState(LexerATNSimulator.SimState settings, ICharStream input, DFAState dfaState)
 {
     settings.index    = input.Index;
     settings.line     = _line;
     settings.charPos  = charPositionInLine;
     settings.dfaState = dfaState;
 }
コード例 #2
0
 protected internal virtual int FailOrAccept(LexerATNSimulator.SimState prevAccept, ICharStream input, ATNConfigSet reach, int t)
 {
     if (prevAccept.dfaState != null)
     {
         LexerActionExecutor lexerActionExecutor = prevAccept.dfaState.LexerActionExecutor;
         Accept(input, lexerActionExecutor, startIndex, prevAccept.index, prevAccept.line, prevAccept.charPos);
         return(prevAccept.dfaState.Prediction);
     }
     else
     {
         // if no accept and EOF is first char, return EOF
         if (t == IntStreamConstants.Eof && input.Index == startIndex)
         {
             return(TokenConstants.Eof);
         }
         throw new LexerNoViableAltException(recog, input, startIndex, reach);
     }
 }