org.millscript.millscript.syntax
Interface Tokenizer

All Superinterfaces:

org.millscript.commons.alert.AlertOrigin

All Known Subinterfaces:

Parser

All Known Implementing Classes:

ConfigurationParserImpl, ParserImpl, SpiceClassParserDelegate, TokenizerImpl

public interface Tokenizer
extends org.millscript.commons.alert.AlertOrigin

This defines the MillScript tokenizer interface.

Method Summary

boolean	checkWhere(char ch) Check if the tokenzier is using the specified strategy for reading names and strings.
void	dropToken() Drops the current token, ready to read the next one.
java.lang.String	getAttributeName() Returns the current token, checking it is a valid XML element attribute name.
java.lang.String	getErrorString() Returns the current token for use as an error message.
int	getInt() Returns the current token as an int.
int	getLineNumber() Returns the line number of the current token.
java.lang.String	getName() Returns the current token as a name, or interned string.
java.lang.String	getOrigin() Returns the origin of this tokenizers character source.
char	getQuoteChar() Returns the first character of the current token, which will be the quote character for a string token.
java.lang.String	getString() Returns the current token as a string.
java.lang.String	getStringNoQuotes() Returns the current token as a string, without the surrounding quotes.
java.lang.String	getStringNoQuotes(int a, int b) Returns the current token as a string, with the specified number of characters removed from the start and end.
java.lang.String	getTagName() Returns the current token, checking it is a valid XML element name.
java.util.regex.Pattern	makePattern() Returns a new tradition regular expression Pattern, made from the contents of the current buffer.
void	markReader() Marks the current position in the reader, so we can jump back to the marked position if required.
void	mustRead(java.lang.String sym) Tests if the next token is the same as the specified string.
TokenType	nextToken() Returns the type of the next token.
boolean	peekRead(java.lang.String sym) Peeks a look at the next token and compares it to the specified value.
TokenType	peekToken() Peeks a look at the type of the next token to be returned by nextToken.
void	resetReader() Resets the reader to the previously marked position.
void	setWhere(char ch) Sets the tokenizers current strategy for reading names and strings.
boolean	tryRead(java.lang.String sym) Tests if the next token is the same as the specified string.

Methods inherited from interface org.millscript.commons.alert.AlertOrigin

setContext, setLineNumber, setOrigin

Method Detail

checkWhere

boolean checkWhere(char ch)

Check if the tokenzier is using the specified strategy for reading names and strings.

Parameters:

ch - the test strategy

Returns:

true if the test and current strategy are the same, otherwise false

dropToken

void dropToken()

Drops the current token, ready to read the next one.

getAttributeName

java.lang.String getAttributeName()

Returns the current token, checking it is a valid XML element attribute name.

Returns:

a string holding the current token, which will be a valid XML element attribute name

getErrorString

java.lang.String getErrorString()

Returns the current token for use as an error message. As such it will be the current token, or a special message if the end of file has been reached.

Returns:

a string holding the current token as an error message

getInt

int getInt()

Returns the current token as an int.

Returns:

the int value of the current token

getLineNumber

int getLineNumber()

Returns the line number of the current token.

Specified by:

getLineNumber in interface org.millscript.commons.alert.AlertOrigin

Returns:

the line number of the current token

getName

java.lang.String getName()

Returns the current token as a name, or interned string.

Returns:

an interned string for the current token

getOrigin

java.lang.String getOrigin()

Returns the origin of this tokenizers character source.

Specified by:

getOrigin in interface org.millscript.commons.alert.AlertOrigin

Returns:

this tokenziers character source origin

getQuoteChar

char getQuoteChar()

Returns the first character of the current token, which will be the quote character for a string token.

Returns:

the first character of the current token

getString

java.lang.String getString()

Returns the current token as a string.

Returns:

a string holding the current token

getStringNoQuotes

java.lang.String getStringNoQuotes()

Returns the current token as a string, without the surrounding quotes.

Returns:

a String with the current token, leading and trailing characters removed

getStringNoQuotes

java.lang.String getStringNoQuotes(int a,
                                   int b)

Returns the current token as a string, with the specified number of characters removed from the start and end.

Parameters:

a - the number of character to remove from the start of the current token

b - the number of character to remove from the end of the current token

Returns:

a String with the current token, with the specified number of characters removed from the start and end

getTagName

java.lang.String getTagName()

Returns the current token, checking it is a valid XML element name.

Returns:

a string holding the current token, which will be a valid XML element name

makePattern

java.util.regex.Pattern makePattern()

Returns a new tradition regular expression Pattern, made from the contents of the current buffer. It looks for the last / in the string to see if there are any flags specified.

Returns:

a new Pattern for the current buffer

markReader

void markReader()

Marks the current position in the reader, so we can jump back to the marked position if required.

mustRead

void mustRead(java.lang.String sym)

Tests if the next token is the same as the specified string. If successful, the token will be dropped, otherwise an alert will be raised.

Parameters:

sym - the token we want to try and read

nextToken

TokenType nextToken()

Returns the type of the next token. It's unlikely that the TokenType.NEED_NEW token will be returned by this method.

Returns:

the TokenType of the next token

peekRead

boolean peekRead(java.lang.String sym)

Peeks a look at the next token and compares it to the specified value.

Parameters:

sym - the token we want to try and read

Returns:

true if the next token is the same as the one specified, false otherwise

peekToken

TokenType peekToken()

Peeks a look at the type of the next token to be returned by nextToken.

Returns:

the TokenType of the next token to be returned by a call to nextToken()

resetReader

void resetReader()

Resets the reader to the previously marked position.

setWhere

void setWhere(char ch)

Sets the tokenizers current strategy for reading names and strings. This is used to let the tokenizer know when it's reading an XML tag name, attribute name or comment.

Parameters:

ch - the new strategy

See Also:

#where

tryRead

boolean tryRead(java.lang.String sym)

Tests if the next token is the same as the specified string. If successful, the token will be dropped.

Parameters:

sym - the token we want to try and read

Returns:

true if the next token is the same as the one specified, false otherwise