Uses of Class
sjm.parse.tokens.Tokenizer

Packages that use Tokenizer

org.jsesoft.jpp	org.jsesoft.jpp: A Java macro preprocessor Title: A Java macro preprocessor.
sjm.parse.tokens

Uses of Tokenizer in org.jsesoft.jpp

Subclasses of Tokenizer in org.jsesoft.jpp

class	MacroTokenizer The tokenizer for the macro processor.
class	Tokenizer Customizable tokenizer fitting Java requirements.

Uses of Tokenizer in sjm.parse.tokens

Fields in sjm.parse.tokens declared as Tokenizer

protected Tokenizer

TokenStringSource.tokenizer

Methods in sjm.parse.tokens with parameters of type Tokenizer

abstract Token	TokenizerState.nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a token that represents a logical piece of a reader.
Token	SlashStarState.nextToken(java.io.PushbackReader r, int theStar, Tokenizer t) Ignore everything up to a closing star and slash, and then return the tokenizer's next token.
Token	NumberState.nextToken(java.io.PushbackReader r, int cin, Tokenizer t) Return a number token from a reader.
protected Token	NumberState.value(java.io.PushbackReader r, Tokenizer t)
Token	QuoteState.nextToken(java.io.PushbackReader r, int cin, Tokenizer t) Return a quoted string token from a reader.
Token	WhitespaceState.nextToken(java.io.PushbackReader r, int aWhitespaceChar, Tokenizer t) Ignore whitespace (such as blanks and tabs), and return the tokenizer's next token.
Token	WordState.nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a word token from a reader.
Token	SlashSlashState.nextToken(java.io.PushbackReader r, int theSlash, Tokenizer t) Ignore everything up to an end-of-line and return the tokenizer's next token.
Token	SlashState.nextToken(java.io.PushbackReader r, int theSlash, Tokenizer t) Either delegate to a comment-handling state, or return a token with just a slash in it.
Token	SymbolState.nextToken(java.io.PushbackReader r, int first, Tokenizer t) Return a symbol token from a reader.

Constructors in sjm.parse.tokens with parameters of type Tokenizer

TokenStringSource(Tokenizer tokenizer, java.lang.String delimiter)
Constructs a TokenStringSource that will read TokenStrings using the specified tokenizer, delimited by the specified delimiter.

TokenAssembly(Tokenizer t)
Constructs a TokenAssembly on a TokenString constructed from the given Tokenizer.

TokenString(Tokenizer t)
Constructs a tokenString from the supplied reader and tokenizer.