sjm.parse.tokens
Class TokenizerState

java.lang.Object
  |
  +--sjm.parse.tokens.TokenizerState
Direct Known Subclasses:
NumberState, QuoteState, SlashSlashState, SlashStarState, SlashState, SymbolState, WhitespaceState, WordState

public abstract class TokenizerState
extends java.lang.Object

A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader. The tokenizer will typically have a character state table that decides which state to use, depending on an initial character. If a single character is insufficient, a state such as SlashState will read a second character, and may delegate to another state, such as SlashStarState. This prospect of delegation is the reason that the nextToken() method has a tokenizer argument.

Version:
1.0
Author:
Steven J. Metsker

Constructor Summary
TokenizerState()
           
 
Method Summary
abstract  Token nextToken(java.io.PushbackReader r, int c, Tokenizer t)
          Return a token that represents a logical piece of a reader.
 
Methods inherited from class java.lang.Object
, clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

TokenizerState

public TokenizerState()
Method Detail

nextToken

public abstract Token nextToken(java.io.PushbackReader r,
                                int c,
                                Tokenizer t)
                         throws java.io.IOException
Return a token that represents a logical piece of a reader.
Parameters:
PushbackReader - a reader to read from
c - the character that a tokenizer used to determine to use this state
Tokenizer - the tokenizer conducting the overall tokenization of the reader
Returns:
a token that represents a logical piece of the reader
Throws:
java.io.IOException - if there is any problem reading