Package redempt.redlex.processing

Class Lexer

java.lang.Object
redempt.redlex.processing.Lexer
Direct Known Subclasses:
DebugLexer
public class Lexer extends Object
A lexer which will tokenize an input String. Best created with BNFParser.createLexer(Path)

  • Constructor Details

    • Lexer

      public Lexer(TokenType root)
      Create a Lexer from a token. Not recommended unless you want to do everything by hand.
      Parameters:
      root - The root TokenType for this Lexer

  • Method Details

    • debug

      public DebugLexer debug()
      Create a DebugLexer to diagnose issues with your bnf. Warning: Irreversibly mutates all tokens, even using this Lexer.
      Returns:
      A DebugLexer which can be used to follow the lexing process and determine where something went wrong

    • getRoot

      public TokenType getRoot()
      Returns:
      The root TokenType for this Lexer

    • getStrategy

      public CullStrategy getStrategy(Token token)

    • setRetainEmpty

      public void setRetainEmpty(boolean retainEmpty)
      Sets whether this Lexer will retain empty tokens
      Parameters:
      retainEmpty - Whether this Lexer should retain empty tokens

    • setRetainStringLiterals

      public void setRetainStringLiterals(boolean retainStringLiterals)
      Sets whether this Lexer will retain string literal tokens
      Parameters:
      retainStringLiterals - Whether this Lexer should retain string literal tokens

    • setUnnamedRule

      public void setUnnamedRule(CullStrategy unnamedRule)
      Sets how this Lexer will handle unnamed tokens
      Parameters:
      unnamedRule - What should be done with unnamed tokens

    • setRuleByName

      public void setRuleByName(CullStrategy strategy, String... names)
      Sets how tokens with given names should be handled
      Parameters:
      strategy - The strategy to use on the tokens with the given names
      names - The names to apply the rule to

    • tokenize

      public Token tokenize(String str, boolean errorOnFail)
      Tokenizes an input String
      Parameters:
      str - The string to tokenize
      errorOnFail - Whether to throw a LexException on failure, returns null if false
      Returns:
      The root Token of the tokenized tree

    • tokenize

      public Token tokenize(String str)
      Tokenizes an input String
      Parameters:
      str - The string to tokenize
      Returns:
      The root Token of the tokenized tree