org.jsesoft.jpp
Class Tokenizer

java.lang.Object
  |
  +--sjm.parse.tokens.Tokenizer
        |
        +--org.jsesoft.jpp.Tokenizer
Direct Known Subclasses:
MacroTokenizer

public class Tokenizer
extends Tokenizer

Customizable tokenizer fitting Java requirements.

Version:
1.0
Author:
JSESoft

Field Summary
(package private)  CommentSlashState commentSlashState
          The commentSlash state.
protected static int DEFAULT_SYMBOL_MAX
           
(package private)  DefaultState defaultState
          The default state.
(package private)  IdentifierState identifierState
          The identifier state.
(package private)  KeywordIdentifierState keywordIdentifierState
          The keyword state.
(package private)  NumberState numberState
          The number state.
(package private)  OperatorState operatorState
          The operator state.
(package private)  QuotedState quotedState
          The quoted state.
(package private)  SeparatorState separatorState
          The separator state.
(package private)  WhitespaceState whitespaceState
          The whitespace state.
 
Fields inherited from class sjm.parse.tokens.Tokenizer
characterState, quoteState, reader, slashState, symbolState, wordState
 
Constructor Summary
Tokenizer()
           
Tokenizer(java.lang.String string)
          Constructs a tokenizer to read from the supplied string.
 
Method Summary
 CommentSlashState getCommentSlashState()
          Returns the commentSlash state of the tokenizer.
 DefaultState getDefaultState()
          Returns the default state of the tokenizer.
 IdentifierState getIdentifierState()
          Returns the identifier state of the tokenizer.
 KeywordIdentifierState getKeywordIdentifierState()
          Returns the keyword state of the tokenizer.
 NumberState getNumberState()
          Returns the number state of the tokenizer.
 OperatorState getOperatorState()
          Returns the operator state of the tokenizer.
 QuotedState getQuotedState()
          Returns the quoted state of the tokenizer.
 SeparatorState getSeparatorState()
          Returns the separator state of the tokenizer.
 WhitespaceState getWhitespaceState()
          Returns the whitespace state of the tokenizer.
 boolean isIdentifierStart(char character)
          Determines if the character may be at the start of an identifier.
 boolean isNumberStart(char character)
          Determines if the character may be at the start of a number.
 boolean isWhitespace(char character)
          Determines if the character is a whitespace.
static void main(java.lang.String[] args)
           
 Token nextToken()
          Scans the next token from the reader.
 void setCommentSlashState(CommentSlashState value)
          Specifies the commentSlash state of the tokenizer.
 void setDefaultState(DefaultState value)
          Specifies the default state of the tokenizer.
 void setIdentifierState(IdentifierState value)
          Specifies the identifier state of the tokenizer.
 void setKeywordIdentifierState(KeywordIdentifierState value)
          Specifies the keyword state of the tokenizer.
 void setNumberState(NumberState value)
          Specifies the number state of the tokenizer.
 void setOperatorState(OperatorState value)
          Specifies the operator state of the tokenizer.
 void setQuotedtState(QuotedState value)
          Specifies the quoted state of the tokenizer.
 void setSeparatorState(SeparatorState value)
          Specifies the separator state of the tokenizer.
 void setString(java.lang.String string)
          Set the string to read from.
 void setWhitespaceState(WhitespaceState value)
          Specifies the whitespace state of the tokenizer.
 
Methods inherited from class sjm.parse.tokens.Tokenizer
getReader, numberState, quoteState, setCharacterState, setReader, setString, slashState, symbolState, whitespaceState, wordState
 
Methods inherited from class java.lang.Object
, clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DEFAULT_SYMBOL_MAX

protected static final int DEFAULT_SYMBOL_MAX

identifierState

IdentifierState identifierState
The identifier state.

operatorState

OperatorState operatorState
The operator state.

separatorState

SeparatorState separatorState
The separator state.

whitespaceState

WhitespaceState whitespaceState
The whitespace state.

numberState

NumberState numberState
The number state.

quotedState

QuotedState quotedState
The quoted state.

keywordIdentifierState

KeywordIdentifierState keywordIdentifierState
The keyword state.

commentSlashState

CommentSlashState commentSlashState
The commentSlash state.

defaultState

DefaultState defaultState
The default state.
Constructor Detail

Tokenizer

public Tokenizer()

Tokenizer

public Tokenizer(java.lang.String string)
Constructs a tokenizer to read from the supplied string.
Parameters:
string - the string to read from
Method Detail

nextToken

public Token nextToken()
                throws java.io.IOException
Scans the next token from the reader.
Overrides:
nextToken in class Tokenizer
Returns:
the next token
Throws:
java.io.IOException - if there is any problem reading

getIdentifierState

public IdentifierState getIdentifierState()
Returns the identifier state of the tokenizer.
Returns:
the identifier state

setIdentifierState

public void setIdentifierState(IdentifierState value)
Specifies the identifier state of the tokenizer.
Parameters:
value - the identifier state

getOperatorState

public OperatorState getOperatorState()
Returns the operator state of the tokenizer.
Returns:
the operator state

setOperatorState

public void setOperatorState(OperatorState value)
Specifies the operator state of the tokenizer.
Parameters:
value - the operator state

getSeparatorState

public SeparatorState getSeparatorState()
Returns the separator state of the tokenizer.
Returns:
the separator state

setSeparatorState

public void setSeparatorState(SeparatorState value)
Specifies the separator state of the tokenizer.
Parameters:
value - the separator state

getWhitespaceState

public WhitespaceState getWhitespaceState()
Returns the whitespace state of the tokenizer.
Returns:
the whitespace state

setWhitespaceState

public void setWhitespaceState(WhitespaceState value)
Specifies the whitespace state of the tokenizer.
Parameters:
value - the whitespace state

getNumberState

public NumberState getNumberState()
Returns the number state of the tokenizer.
Returns:
the number state

setNumberState

public void setNumberState(NumberState value)
Specifies the number state of the tokenizer.
Parameters:
value - the number state

getQuotedState

public QuotedState getQuotedState()
Returns the quoted state of the tokenizer.
Returns:
the quoted state

setQuotedtState

public void setQuotedtState(QuotedState value)
Specifies the quoted state of the tokenizer.
Parameters:
value - the quoted state

getKeywordIdentifierState

public KeywordIdentifierState getKeywordIdentifierState()
Returns the keyword state of the tokenizer.
Returns:
the keyword state

setKeywordIdentifierState

public void setKeywordIdentifierState(KeywordIdentifierState value)
Specifies the keyword state of the tokenizer.
Parameters:
value - the keyword state

getCommentSlashState

public CommentSlashState getCommentSlashState()
Returns the commentSlash state of the tokenizer.
Returns:
the commentSlash state

setCommentSlashState

public void setCommentSlashState(CommentSlashState value)
Specifies the commentSlash state of the tokenizer.
Parameters:
value - the commentSlash state

getDefaultState

public DefaultState getDefaultState()
Returns the default state of the tokenizer.
Returns:
the default state

setDefaultState

public void setDefaultState(DefaultState value)
Specifies the default state of the tokenizer.
Parameters:
value - the default state

isIdentifierStart

public boolean isIdentifierStart(char character)
Determines if the character may be at the start of an identifier.
Parameters:
character - the character to check
Returns:
true of character may be at the start of an identifer
See Also:
Character.isJavaIdentifierStart(char)

isWhitespace

public boolean isWhitespace(char character)
Determines if the character is a whitespace.
Parameters:
character - the character to check
Returns:
true of character is a whitespace
See Also:
Character.isWhitespace(char)

isNumberStart

public boolean isNumberStart(char character)
Determines if the character may be at the start of a number.
Parameters:
character - the character to check
Returns:
true of character may be at the start of a number
See Also:
Character.isDigit(char)

setString

public void setString(java.lang.String string)
Set the string to read from.
Overrides:
setString in class Tokenizer
Parameters:
string - the string to read from

main

public static void main(java.lang.String[] args)