Uses of Class
sjm.parse.tokens.TokenizerState

Packages that use TokenizerState

sjm.parse.tokens

Uses of TokenizerState in sjm.parse.tokens

Subclasses of TokenizerState in sjm.parse.tokens

class	NumberState A NumberState object returns a number from a reader.
class	QuoteState A quoteState returns a quoted string token from a reader.
class	SlashSlashState A slashSlash state ignores everything up to an end-of-line and returns the tokenizer's next token.
class	SlashStarState A slashStar state ignores everything up to a closing star and slash, and then returns the tokenizer's next token.
class	SlashState This state will either delegate to a comment-handling state, or return a token with just a slash in it.
class	SymbolState The idea of a symbol is a character that stands on its own, such as an ampersand or a parenthesis.
class	WhitespaceState A whitespace state ignores whitespace (such as blanks and tabs), and returns the tokenizer's next token.
class	WordState A wordState returns a word from a reader.

Fields in sjm.parse.tokens declared as TokenizerState

protected TokenizerState[]

Tokenizer.characterState

Methods in sjm.parse.tokens with parameters of type TokenizerState

void	Tokenizer.setCharacterState(int from, int to, TokenizerState state) Change the state the tokenizer will enter upon reading any character between "from" and "to".