sjm.parse.tokens
Class TokenizerState
java.lang.Object
|
+--sjm.parse.tokens.TokenizerState
- Direct Known Subclasses:
- NumberState, QuoteState, SlashSlashState, SlashStarState, SlashState, SymbolState, WhitespaceState, WordState
- public abstract class TokenizerState
- extends java.lang.Object
A tokenizerState returns a token, given a reader, an
initial character read from the reader, and a tokenizer
that is conducting an overall tokenization of the reader.
The tokenizer will typically have a character state table
that decides which state to use, depending on an initial
character. If a single character is insufficient, a state
such as SlashState
will read a second
character, and may delegate to another state, such as
SlashStarState
. This prospect of delegation is
the reason that the nextToken()
method has a
tokenizer argument.
- Version:
- 1.0
- Author:
- Steven J. Metsker
Method Summary |
abstract Token |
nextToken(java.io.PushbackReader r,
int c,
Tokenizer t)
Return a token that represents a logical piece of a reader. |
Methods inherited from class java.lang.Object |
, clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
TokenizerState
public TokenizerState()
nextToken
public abstract Token nextToken(java.io.PushbackReader r,
int c,
Tokenizer t)
throws java.io.IOException
- Return a token that represents a logical piece of a reader.
- Parameters:
PushbackReader
- a reader to read fromc
- the character that a tokenizer used to
determine to use this stateTokenizer
- the tokenizer conducting the overall
tokenization of the reader- Returns:
- a token that represents a logical piece of the
reader
- Throws:
java.io.IOException
- if there is any problem reading