Package sjm.parse.tokens

Class Summary

CaselessLiteral	A CaselessLiteral matches a specified String from an assembly, disregarding case.
Literal	A Literal matches a specific String from an assembly.
Num	A Num matches a number from a token assembly.
NumberState	A NumberState object returns a number from a reader.
QuotedString	A QuotedString matches a quoted string, like "this one" from a token assembly.
QuoteState	A quoteState returns a quoted string token from a reader.
SlashSlashState	A slashSlash state ignores everything up to an end-of-line and returns the tokenizer's next token.
SlashStarState	A slashStar state ignores everything up to a closing star and slash, and then returns the tokenizer's next token.
SlashState	This state will either delegate to a comment-handling state, or return a token with just a slash in it.
Symbol	A Symbol matches a specific sequence, such as <, or <= that a tokenizer returns as a symbol.
SymbolNode	A SymbolNode object is a member of a tree that contains all possible prefixes of allowable symbols.
SymbolRootNode	This class is a special case of a SymbolNode.
SymbolState	The idea of a symbol is a character that stands on its own, such as an ampersand or a parenthesis.
Token	A token represents a logical chunk of a string.
TokenAssembly	A TokenAssembly is an Assembly whose elements are Tokens.
Tokenizer	A tokenizer divides a string into tokens.
TokenizerState	A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader.
TokenString	A TokenString is like a String, but it is a series of Tokens rather than a series of chars.
TokenStringSource	A TokenStringSource enumerates over a specified reader, returning TokenStrings delimited by a specified delimiter.
TokenTester
TokenType	Objects of this class represent a type of token, such as "number" or "word".
WhitespaceState	A whitespace state ignores whitespace (such as blanks and tabs), and returns the tokenizer's next token.
Word	A Word matches a word from a token assembly.
WordState	A wordState returns a word from a reader.