Package redempt.redlex.data

Class TokenType

java.lang.Object

redempt.redlex.data.TokenType

Direct Known Subclasses:

CharGroupToken, CharSetToken, ChoiceToken, DebugToken, EndOfFileToken, ListToken, NotToken, PlaceholderToken, RepeatingToken, StringChoiceToken, StringToken, WrapperToken

public abstract class TokenType
extends Object

Represents a type of token which can be used to create a Lexer

Constructor Summary

Constructors

Constructor	Description
TokenType(String name)	Create a new TokenType with the given name

Method Summary

Modifier and Type	Method	Description
protected abstract List<Character>	calcFirstCharacters()
protected abstract Token	findForward(String str, int pos, LexContext ctx)
List<Character>	getFirstCharacters()
int	getId()
Lexer	getLexer()
String	getMessage()
String	getName()
abstract int	minLength()
boolean	nameMatches(String name)	Check whether the name of this TokenType matches the given name
void	replacePlaceholders(Map<String,TokenType> tokens)
void	setLexer(Lexer lexer)	Sets the Lexer this TokenType and its children belong to
void	setName(String name)	Sets the name of this TokenType
String	toString()
Token	tryTokenize(String str, int pos, LexContext ctx)
void	walk(Consumer<TokenType> each)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

TokenType

public TokenType(String name)

Create a new TokenType with the given name

Parameters:

name - The name for this TokenType

Method Details

getId

public int getId()

Returns:

The numeric ID of this TokenType

setLexer

public void setLexer(Lexer lexer)

Sets the Lexer this TokenType and its children belong to

Parameters:

lexer - The Lexer

getLexer

public Lexer getLexer()

Returns:

The Lexer this TokenType belongs to

nameMatches

public boolean nameMatches(String name)

Check whether the name of this TokenType matches the given name

Parameters:

name - The name to compare

Returns:

Whether this TokenType's name matches the given name

getName

public String getName()

Returns:

The name of this TokenType

setName

public void setName(String name)

Sets the name of this TokenType

Parameters:

name - The name to set

tryTokenize

public Token tryTokenize(String str,
 int pos,
 LexContext ctx)

getMessage

public String getMessage()

replacePlaceholders

public void replacePlaceholders(Map<String,TokenType> tokens)

walk

public void walk(Consumer<TokenType> each)

findForward

protected abstract Token findForward(String str,
 int pos,
 LexContext ctx)

minLength

public abstract int minLength()

calcFirstCharacters

protected abstract List<Character> calcFirstCharacters()

getFirstCharacters

public List<Character> getFirstCharacters()

Returns:

A list of all the characters which may appear as the first character of this token. Includes null if this token may be zero-length.

toString

public String toString()

Overrides:

toString in class Object