sjm.parse.tokens
Class TokenizerState

java.lang.Object
  |
  +--sjm.parse.tokens.TokenizerState

Direct Known Subclasses:

NumberState, QuoteState, SlashSlashState, SlashStarState, SlashState, SymbolState, WhitespaceState, WordState

public abstract class TokenizerState

extends java.lang.Object

A tokenizerState returns a token, given a reader, an initial character read from the reader, and a tokenizer that is conducting an overall tokenization of the reader. The tokenizer will typically have a character state table that decides which state to use, depending on an initial character. If a single character is insufficient, a state such as SlashState will read a second character, and may delegate to another state, such as SlashStarState. This prospect of delegation is the reason that the nextToken() method has a tokenizer argument.

Version:

1.0

Author:

Steven J. Metsker

Constructor Summary

TokenizerState()

Method Summary

abstract Token

nextToken(java.io.PushbackReader r, int c, Tokenizer t) Return a token that represents a logical piece of a reader.

Methods inherited from class java.lang.Object

, clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerState

public TokenizerState()

Method Detail

nextToken

public abstract Token nextToken(java.io.PushbackReader r,
                                int c,
                                Tokenizer t)
                         throws java.io.IOException

Return a token that represents a logical piece of a reader.

Parameters:

PushbackReader - a reader to read from

c - the character that a tokenizer used to determine to use this state

Tokenizer - the tokenizer conducting the overall tokenization of the reader

Returns:

a token that represents a logical piece of the reader

Throws:

java.io.IOException - if there is any problem reading