org.millscript.millscript.syntax
Class TokenizerImpl

java.lang.Object
  org.millscript.millscript.syntax.TokenizerImpl

All Implemented Interfaces:

org.millscript.commons.alert.AlertOrigin, Tokenizer

Direct Known Subclasses:

ConfigurationParserImpl

public class TokenizerImpl
extends java.lang.Object
implements Tokenizer

This class performs the basic tokenisation of the MillScript language. A single instance of this class would be used to tokenise one file/source.

Field Summary

protected Configuration	config The current configuration.
protected org.millscript.commons.util.map.EHashMap<java.lang.Integer,java.lang.CharSequence>	interpolationMap Used to store interpolation values, mapping the position in the interpolated string to the value to be interpolated.

Method Summary

boolean	checkWhere(char ch) Check if the tokenzier is using the specified strategy for reading names and strings.
void	dropToken() Drops the current token, ready to read the next one.
java.lang.String	getAttributeName() Returns the current token, checking it is a valid XML element attribute name.
java.lang.String	getErrorString() Returns the current token for use as an error message.
int	getInt() Returns the current token as an int.
int	getLineNumber() Returns the line number of the current token.
java.lang.String	getName() Returns the current token as a name, or interned string.
java.lang.String	getOrigin() Returns the origin of this tokenizers character source.
char	getQuoteChar() Returns the first character of the current token, which will be the quote character for a string token.
java.lang.String	getString() Returns the current token as a string.
java.lang.String	getStringNoQuotes() Returns the current token as a string, without the surrounding quotes.
java.lang.String	getStringNoQuotes(int a, int b) Returns the current token as a string, with the specified number of characters removed from the start and end.
java.lang.String	getTagName() Returns the current token, checking it is a valid XML element name.
java.util.regex.Pattern	makePattern() Returns a new tradition regular expression Pattern, made from the contents of the current buffer.
void	markReader() Marks the current position in the reader, so we can jump back to the marked position if required.
void	mustRead(java.lang.String sym) Tests if the next token is the same as the specified string.
TokenType	nextToken() Returns the type of the next token.
boolean	peekRead(java.lang.String sym) Peeks a look at the next token and compares it to the specified value.
TokenType	peekToken() Peeks a look at the type of the next token to be returned by nextToken.
void	resetReader() Resets the reader to the previously marked position.
void	setContext(java.lang.String s, int n)
void	setLineNumber(int n)
void	setOrigin(java.lang.String o)
void	setWhere(char ch) Sets the tokenizers current strategy for reading names and strings.
boolean	tryRead(java.lang.String sym) Tests if the next token is the same as the specified string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

config

protected Configuration config

The current configuration.

interpolationMap

protected org.millscript.commons.util.map.EHashMap<java.lang.Integer,java.lang.CharSequence> interpolationMap

Used to store interpolation values, mapping the position in the interpolated string to the value to be interpolated.

Method Detail

getOrigin

public java.lang.String getOrigin()

Description copied from interface: Tokenizer

Returns the origin of this tokenizers character source.

Specified by:

getOrigin in interface org.millscript.commons.alert.AlertOrigin

Specified by:

getOrigin in interface Tokenizer

Returns:

this tokenziers character source origin

See Also:

Tokenizer.getOrigin()

getLineNumber

public int getLineNumber()

Description copied from interface: Tokenizer

Returns the line number of the current token.

Specified by:

getLineNumber in interface org.millscript.commons.alert.AlertOrigin

Specified by:

getLineNumber in interface Tokenizer

Returns:

the line number of the current token

See Also:

Tokenizer.getLineNumber()

setWhere

public void setWhere(char ch)

Description copied from interface: Tokenizer

Sets the tokenizers current strategy for reading names and strings. This is used to let the tokenizer know when it's reading an XML tag name, attribute name or comment.

Specified by:

setWhere in interface Tokenizer

Parameters:

ch - the new strategy

See Also:

Tokenizer.setWhere(char)

checkWhere

public boolean checkWhere(char ch)

Description copied from interface: Tokenizer

Check if the tokenzier is using the specified strategy for reading names and strings.

Specified by:

checkWhere in interface Tokenizer

Parameters:

ch - the test strategy

Returns:

true if the test and current strategy are the same, otherwise false

See Also:

Tokenizer.checkWhere(char)

makePattern

public java.util.regex.Pattern makePattern()

Description copied from interface: Tokenizer

Returns a new tradition regular expression Pattern, made from the contents of the current buffer. It looks for the last / in the string to see if there are any flags specified.

Specified by:

makePattern in interface Tokenizer

Returns:

a new Pattern for the current buffer

See Also:

Tokenizer.makePattern()

nextToken

public TokenType nextToken()

Description copied from interface: Tokenizer

Returns the type of the next token. It's unlikely that the TokenType.NEED_NEW token will be returned by this method.

Specified by:

nextToken in interface Tokenizer

Returns:

the TokenType of the next token

See Also:

Tokenizer.nextToken()

peekToken

public TokenType peekToken()

Description copied from interface: Tokenizer

Peeks a look at the type of the next token to be returned by nextToken.

Specified by:

peekToken in interface Tokenizer

Returns:

the TokenType of the next token to be returned by a call to Tokenizer.nextToken()

See Also:

Tokenizer.peekToken()

dropToken

public void dropToken()

Description copied from interface: Tokenizer

Drops the current token, ready to read the next one.

Specified by:

dropToken in interface Tokenizer

See Also:

Tokenizer.dropToken()

tryRead

public boolean tryRead(java.lang.String sym)

Description copied from interface: Tokenizer

Tests if the next token is the same as the specified string. If successful, the token will be dropped.

Specified by:

tryRead in interface Tokenizer

Parameters:

sym - the token we want to try and read

Returns:

true if the next token is the same as the one specified, false otherwise

See Also:

Tokenizer.tryRead(java.lang.String)

markReader

public void markReader()

Description copied from interface: Tokenizer

Marks the current position in the reader, so we can jump back to the marked position if required.

Specified by:

markReader in interface Tokenizer

See Also:

Tokenizer.markReader()

resetReader

public void resetReader()

Description copied from interface: Tokenizer

Resets the reader to the previously marked position.

Specified by:

resetReader in interface Tokenizer

See Also:

Tokenizer.resetReader()

peekRead

public boolean peekRead(java.lang.String sym)

Description copied from interface: Tokenizer

Peeks a look at the next token and compares it to the specified value.

Specified by:

peekRead in interface Tokenizer

Parameters:

sym - the token we want to try and read

Returns:

true if the next token is the same as the one specified, false otherwise

See Also:

Tokenizer.peekRead(java.lang.String)

mustRead

public void mustRead(java.lang.String sym)

Description copied from interface: Tokenizer

Tests if the next token is the same as the specified string. If successful, the token will be dropped, otherwise an alert will be raised.

Specified by:

mustRead in interface Tokenizer

Parameters:

sym - the token we want to try and read

See Also:

Tokenizer.mustRead(java.lang.String)

getString

public java.lang.String getString()

Description copied from interface: Tokenizer

Returns the current token as a string.

Specified by:

getString in interface Tokenizer

Returns:

a string holding the current token

See Also:

Tokenizer.getString()

getErrorString

public java.lang.String getErrorString()

Description copied from interface: Tokenizer

Returns the current token for use as an error message. As such it will be the current token, or a special message if the end of file has been reached.

Specified by:

getErrorString in interface Tokenizer

Returns:

a string holding the current token as an error message

See Also:

Tokenizer.getErrorString()

getAttributeName

public java.lang.String getAttributeName()

Description copied from interface: Tokenizer

Returns the current token, checking it is a valid XML element attribute name.

Specified by:

getAttributeName in interface Tokenizer

Returns:

a string holding the current token, which will be a valid XML element attribute name

See Also:

Tokenizer.getAttributeName()

getTagName

public java.lang.String getTagName()

Description copied from interface: Tokenizer

Returns the current token, checking it is a valid XML element name.

Specified by:

getTagName in interface Tokenizer

Returns:

a string holding the current token, which will be a valid XML element name

See Also:

Tokenizer.getTagName()

getStringNoQuotes

public java.lang.String getStringNoQuotes()

Description copied from interface: Tokenizer

Returns the current token as a string, without the surrounding quotes.

Specified by:

getStringNoQuotes in interface Tokenizer

Returns:

a String with the current token, leading and trailing characters removed

See Also:

Tokenizer.getStringNoQuotes()

getStringNoQuotes

public java.lang.String getStringNoQuotes(int a,
                                          int b)

Description copied from interface: Tokenizer

Returns the current token as a string, with the specified number of characters removed from the start and end.

Specified by:

getStringNoQuotes in interface Tokenizer

Parameters:

a - the number of character to remove from the start of the current token

b - the number of character to remove from the end of the current token

Returns:

a String with the current token, with the specified number of characters removed from the start and end

See Also:

Tokenizer.getStringNoQuotes(int, int)

getQuoteChar

public char getQuoteChar()

Description copied from interface: Tokenizer

Returns the first character of the current token, which will be the quote character for a string token.

Specified by:

getQuoteChar in interface Tokenizer

Returns:

the first character of the current token

See Also:

Tokenizer.getQuoteChar()

getName

public java.lang.String getName()

Description copied from interface: Tokenizer

Returns the current token as a name, or interned string.

Specified by:

getName in interface Tokenizer

Returns:

an interned string for the current token

See Also:

Tokenizer.getName()

getInt

public int getInt()

Description copied from interface: Tokenizer

Returns the current token as an int.

Specified by:

getInt in interface Tokenizer

Returns:

the int value of the current token

See Also:

Tokenizer.getInt()

setContext

public void setContext(java.lang.String s,
                       int n)

Specified by:

setContext in interface org.millscript.commons.alert.AlertOrigin

See Also:

AlertOrigin.setContext(java.lang.String, int)

setLineNumber

public void setLineNumber(int n)

Specified by:

setLineNumber in interface org.millscript.commons.alert.AlertOrigin

See Also:

AlertOrigin.setLineNumber(int)

setOrigin

public void setOrigin(java.lang.String o)

Specified by:

setOrigin in interface org.millscript.commons.alert.AlertOrigin

See Also:

AlertOrigin.setOrigin(java.lang.String)