sjm.parse.tokens
Class SymbolState

java.lang.Object
  |
  +--sjm.parse.tokens.TokenizerState
        |
        +--sjm.parse.tokens.SymbolState

public class SymbolState

extends TokenizerState

The idea of a symbol is a character that stands on its own, such as an ampersand or a parenthesis. For example, when tokenizing the expression (isReady)& (isWilling) , a typical tokenizer would return 7 tokens, including one for each parenthesis and one for the ampersand. Thus a series of symbols such as )&( becomes three tokens, while a series of letters such as isReady becomes a single word token.

Multi-character symbols are an exception to the rule that a symbol is a standalone character. For example, a tokenizer may want less-than-or-equals to tokenize as a single token. This class provides a method for establishing which multi-character symbols an object of this class should treat as single symbols. This allows, for example, "cat <= dog" to tokenize as three tokens, rather than splitting the less-than and equals symbols into separate tokens.

By default, this state recognizes the following multi- character symbols: !=, :-, <=, >=

Version:

1.0

Author:

Steven J. Metsker

Field Summary

(package private) SymbolRootNode

symbols

Constructor Summary

SymbolState()
Constructs a symbol state with a default idea of what multi-character symbols to accept (as described in the class comment).

Method Summary

void	add(java.lang.String s) Add a multi-character symbol.
Token	nextToken(java.io.PushbackReader r, int first, Tokenizer t) Return a symbol token from a reader.

Methods inherited from class java.lang.Object

, clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

symbols

SymbolRootNode symbols

Constructor Detail

SymbolState

public SymbolState()

Constructs a symbol state with a default idea of what multi-character symbols to accept (as described in the class comment).

Method Detail

add

public void add(java.lang.String s)

Add a multi-character symbol.

Parameters:

String - the symbol to add, such as "=:="

nextToken

public Token nextToken(java.io.PushbackReader r,
                       int first,
                       Tokenizer t)
                throws java.io.IOException

Return a symbol token from a reader.

Overrides:

nextToken in class TokenizerState

Returns:

a symbol token from a reader