org.apache.commons.lang.text

Class StrTokenizer

public class StrTokenizer extends Object implements ListIterator, Cloneable

Tokenizes a string based based on delimiters (separators) and supporting quoting and ignored character concepts.

This class can split a String into many smaller strings. It aims to do a similar job to java.util.StringTokenizer StringTokenizer, however it offers much more control and flexibility including implementing the ListIterator interface. By default, it is set up like StringTokenizer.

The input String is split into a number of tokens. Each token is separated from the next String by a delimiter. One or more delimiter characters must be specified.

Each token may be surrounded by quotes. The quote matcher specifies the quote character(s). A quote may be escaped within a quoted section by duplicating itself.

Between each token and the delimiter are potentially characters that need trimming. The trimmer matcher specifies these characters. One usage might be to trim whitespace characters.

At any point outside the quotes there might potentially be invalid characters. The ignored matcher specifies these characters to be removed. One usage might be to remove new line characters.

Empty tokens may be removed or returned as null.

 "a,b,c"         - Three tokens "a","b","c"   (comma delimiter)
 " a, b , c "    - Three tokens "a","b","c"   (default CSV processing trims whitespace)
 "a, ", b ,", c" - Three tokens "a, " , " b ", ", c" (quoted text untouched)
 

This tokenizer has the following properties and options:

PropertyTypeDefault
delimCharSetMatcher{ \t\n\r\f}
quoteNoneMatcher{}
ignoreNoneMatcher{}
emptyTokenAsNullbooleanfalse
ignoreEmptyTokensbooleantrue

Since: 2.2

Version: $Id: StrTokenizer.java 592077 2007-11-05 16:47:10Z mbenson $

Author: Matthew Inger Stephen Colebourne Gary D. Gregory

Constructor Summary
StrTokenizer()
Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer, but with no text to tokenize.
StrTokenizer(String input)
Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.
StrTokenizer(String input, char delim)
Constructs a tokenizer splitting on the specified delimiter character.
StrTokenizer(String input, String delim)
Constructs a tokenizer splitting on the specified delimiter string.
StrTokenizer(String input, StrMatcher delim)
Constructs a tokenizer splitting using the specified delimiter matcher.
StrTokenizer(String input, char delim, char quote)
Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
StrTokenizer(String input, StrMatcher delim, StrMatcher quote)
Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
StrTokenizer(char[] input)
Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.
StrTokenizer(char[] input, char delim)
Constructs a tokenizer splitting on the specified character.
StrTokenizer(char[] input, String delim)
Constructs a tokenizer splitting on the specified string.
StrTokenizer(char[] input, StrMatcher delim)
Constructs a tokenizer splitting using the specified delimiter matcher.
StrTokenizer(char[] input, char delim, char quote)
Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.
StrTokenizer(char[] input, StrMatcher delim, StrMatcher quote)
Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.
Method Summary
voidadd(Object obj)
Unsupported ListIterator operation.
Objectclone()
Creates a new instance of this Tokenizer.
StringgetContent()
Gets the String content that the tokenizer is parsing.
static StrTokenizergetCSVInstance()
Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
static StrTokenizergetCSVInstance(String input)
Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
static StrTokenizergetCSVInstance(char[] input)
Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input.
StrMatchergetDelimiterMatcher()
Gets the field delimiter matcher.
StrMatchergetIgnoredMatcher()
Gets the ignored character matcher.
StrMatchergetQuoteMatcher()
Gets the quote matcher currently in use.
String[]getTokenArray()
Gets a copy of the full token list as an independent modifiable array.
ListgetTokenList()
Gets a copy of the full token list as an independent modifiable list.
StrMatchergetTrimmerMatcher()
Gets the trimmer character matcher.
static StrTokenizergetTSVInstance()
Gets a new tokenizer instance which parses Tab Separated Value strings.
static StrTokenizergetTSVInstance(String input)
Gets a new tokenizer instance which parses Tab Separated Value strings.
static StrTokenizergetTSVInstance(char[] input)
Gets a new tokenizer instance which parses Tab Separated Value strings.
booleanhasNext()
Checks whether there are any more tokens.
booleanhasPrevious()
Checks whether there are any previous tokens that can be iterated to.
booleanisEmptyTokenAsNull()
Gets whether the tokenizer currently returns empty tokens as null.
booleanisIgnoreEmptyTokens()
Gets whether the tokenizer currently ignores empty tokens.
Objectnext()
Gets the next token.
intnextIndex()
Gets the index of the next token to return.
StringnextToken()
Gets the next token from the String.
Objectprevious()
Gets the token previous to the last returned token.
intpreviousIndex()
Gets the index of the previous token.
StringpreviousToken()
Gets the previous token from the String.
voidremove()
Unsupported ListIterator operation.
StrTokenizerreset()
Resets this tokenizer, forgetting all parsing and iteration already completed.
StrTokenizerreset(String input)
Reset this tokenizer, giving it a new input string to parse.
StrTokenizerreset(char[] input)
Reset this tokenizer, giving it a new input string to parse.
voidset(Object obj)
Unsupported ListIterator operation.
StrTokenizersetDelimiterChar(char delim)
Sets the field delimiter character.
StrTokenizersetDelimiterMatcher(StrMatcher delim)
Sets the field delimiter matcher.
StrTokenizersetDelimiterString(String delim)
Sets the field delimiter string.
StrTokenizersetEmptyTokenAsNull(boolean emptyAsNull)
Sets whether the tokenizer should return empty tokens as null.
StrTokenizersetIgnoredChar(char ignored)
Set the character to ignore.
StrTokenizersetIgnoredMatcher(StrMatcher ignored)
Set the matcher for characters to ignore.
StrTokenizersetIgnoreEmptyTokens(boolean ignoreEmptyTokens)
Sets whether the tokenizer should ignore and not return empty tokens.
StrTokenizersetQuoteChar(char quote)
Sets the quote character to use.
StrTokenizersetQuoteMatcher(StrMatcher quote)
Set the quote matcher to use.
StrTokenizersetTrimmerMatcher(StrMatcher trimmer)
Sets the matcher for characters to trim.
intsize()
Gets the number of tokens found in the String.
protected Listtokenize(char[] chars, int offset, int count)
Internal method to performs the tokenization.
StringtoString()
Gets the String content that the tokenizer is parsing.

Constructor Detail

StrTokenizer

public StrTokenizer()
Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer, but with no text to tokenize.

This constructor is normally used with reset.

StrTokenizer

public StrTokenizer(String input)
Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.

Parameters: input the string which is to be parsed

StrTokenizer

public StrTokenizer(String input, char delim)
Constructs a tokenizer splitting on the specified delimiter character.

Parameters: input the string which is to be parsed delim the field delimiter character

StrTokenizer

public StrTokenizer(String input, String delim)
Constructs a tokenizer splitting on the specified delimiter string.

Parameters: input the string which is to be parsed delim the field delimiter string

StrTokenizer

public StrTokenizer(String input, StrMatcher delim)
Constructs a tokenizer splitting using the specified delimiter matcher.

Parameters: input the string which is to be parsed delim the field delimiter matcher

StrTokenizer

public StrTokenizer(String input, char delim, char quote)
Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

Parameters: input the string which is to be parsed delim the field delimiter character quote the field quoted string character

StrTokenizer

public StrTokenizer(String input, StrMatcher delim, StrMatcher quote)
Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

Parameters: input the string which is to be parsed delim the field delimiter matcher quote the field quoted string matcher

StrTokenizer

public StrTokenizer(char[] input)
Constructs a tokenizer splitting on space, tab, newline and formfeed as per StringTokenizer.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the string which is to be parsed, not cloned

StrTokenizer

public StrTokenizer(char[] input, char delim)
Constructs a tokenizer splitting on the specified character.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the string which is to be parsed, not cloned delim the field delimiter character

StrTokenizer

public StrTokenizer(char[] input, String delim)
Constructs a tokenizer splitting on the specified string.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the string which is to be parsed, not cloned delim the field delimiter string

StrTokenizer

public StrTokenizer(char[] input, StrMatcher delim)
Constructs a tokenizer splitting using the specified delimiter matcher.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the string which is to be parsed, not cloned delim the field delimiter matcher

StrTokenizer

public StrTokenizer(char[] input, char delim, char quote)
Constructs a tokenizer splitting on the specified delimiter character and handling quotes using the specified quote character.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the string which is to be parsed, not cloned delim the field delimiter character quote the field quoted string character

StrTokenizer

public StrTokenizer(char[] input, StrMatcher delim, StrMatcher quote)
Constructs a tokenizer splitting using the specified delimiter matcher and handling quotes using the specified quote matcher.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the string which is to be parsed, not cloned delim the field delimiter character quote the field quoted string character

Method Detail

add

public void add(Object obj)
Unsupported ListIterator operation.

Parameters: obj this parameter ignored.

Throws: UnsupportedOperationException always

clone

public Object clone()
Creates a new instance of this Tokenizer. The new instance is reset so that it will be at the start of the token list. If a CloneNotSupportedException is caught, return null.

Returns: a new instance of this Tokenizer which has been reset.

getContent

public String getContent()
Gets the String content that the tokenizer is parsing.

Returns: the string content being parsed

getCSVInstance

public static StrTokenizer getCSVInstance()
Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

You must call a "reset" method to set the string which you want to parse.

Returns: a new tokenizer instance which parses Comma Separated Value strings

getCSVInstance

public static StrTokenizer getCSVInstance(String input)
Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

Parameters: input the text to parse

Returns: a new tokenizer instance which parses Comma Separated Value strings

getCSVInstance

public static StrTokenizer getCSVInstance(char[] input)
Gets a new tokenizer instance which parses Comma Separated Value strings initializing it with the given input. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

Parameters: input the text to parse

Returns: a new tokenizer instance which parses Comma Separated Value strings

getDelimiterMatcher

public StrMatcher getDelimiterMatcher()
Gets the field delimiter matcher.

Returns: the delimiter matcher in use

getIgnoredMatcher

public StrMatcher getIgnoredMatcher()
Gets the ignored character matcher.

These characters are ignored when parsing the String, unless they are within a quoted region. The default value is not to ignore anything.

Returns: the ignored matcher in use

getQuoteMatcher

public StrMatcher getQuoteMatcher()
Gets the quote matcher currently in use.

The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data. The default value is '"' (double quote).

Returns: the quote matcher in use

getTokenArray

public String[] getTokenArray()
Gets a copy of the full token list as an independent modifiable array.

Returns: the tokens as a String array

getTokenList

public List getTokenList()
Gets a copy of the full token list as an independent modifiable list.

Returns: the tokens as a String array

getTrimmerMatcher

public StrMatcher getTrimmerMatcher()
Gets the trimmer character matcher.

These characters are trimmed off on each side of the delimiter until the token or quote is found. The default value is not to trim anything.

Returns: the trimmer matcher in use

getTSVInstance

public static StrTokenizer getTSVInstance()
Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

You must call a "reset" method to set the string which you want to parse.

Returns: a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance

public static StrTokenizer getTSVInstance(String input)
Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

Parameters: input the string to parse

Returns: a new tokenizer instance which parses Tab Separated Value strings.

getTSVInstance

public static StrTokenizer getTSVInstance(char[] input)
Gets a new tokenizer instance which parses Tab Separated Value strings. The default for CSV processing will be trim whitespace from both ends (which can be overridden with the setTrimmer method).

Parameters: input the string to parse

Returns: a new tokenizer instance which parses Tab Separated Value strings.

hasNext

public boolean hasNext()
Checks whether there are any more tokens.

Returns: true if there are more tokens

hasPrevious

public boolean hasPrevious()
Checks whether there are any previous tokens that can be iterated to.

Returns: true if there are previous tokens

isEmptyTokenAsNull

public boolean isEmptyTokenAsNull()
Gets whether the tokenizer currently returns empty tokens as null. The default for this property is false.

Returns: true if empty tokens are returned as null

isIgnoreEmptyTokens

public boolean isIgnoreEmptyTokens()
Gets whether the tokenizer currently ignores empty tokens. The default for this property is true.

Returns: true if empty tokens are not returned

next

public Object next()
Gets the next token. This method is equivalent to nextToken.

Returns: the next String token

nextIndex

public int nextIndex()
Gets the index of the next token to return.

Returns: the next token index

nextToken

public String nextToken()
Gets the next token from the String.

Returns: the next sequential token, or null when no more tokens are found

previous

public Object previous()
Gets the token previous to the last returned token.

Returns: the previous token

previousIndex

public int previousIndex()
Gets the index of the previous token.

Returns: the previous token index

previousToken

public String previousToken()
Gets the previous token from the String.

Returns: the previous sequential token, or null when no more tokens are found

remove

public void remove()
Unsupported ListIterator operation.

Throws: UnsupportedOperationException always

reset

public StrTokenizer reset()
Resets this tokenizer, forgetting all parsing and iteration already completed.

This method allows the same tokenizer to be reused for the same String.

Returns: this, to enable chaining

reset

public StrTokenizer reset(String input)
Reset this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.

Parameters: input the new string to tokenize, null sets no text to parse

Returns: this, to enable chaining

reset

public StrTokenizer reset(char[] input)
Reset this tokenizer, giving it a new input string to parse. In this manner you can re-use a tokenizer with the same settings on multiple input lines.

The input character array is not cloned, and must not be altered after passing in to this method.

Parameters: input the new character array to tokenize, not cloned, null sets no text to parse

Returns: this, to enable chaining

set

public void set(Object obj)
Unsupported ListIterator operation.

Parameters: obj this parameter ignored.

Throws: UnsupportedOperationException always

setDelimiterChar

public StrTokenizer setDelimiterChar(char delim)
Sets the field delimiter character.

Parameters: delim the delimiter character to use

Returns: this, to enable chaining

setDelimiterMatcher

public StrTokenizer setDelimiterMatcher(StrMatcher delim)
Sets the field delimiter matcher.

The delimitier is used to separate one token from another.

Parameters: delim the delimiter matcher to use

Returns: this, to enable chaining

setDelimiterString

public StrTokenizer setDelimiterString(String delim)
Sets the field delimiter string.

Parameters: delim the delimiter string to use

Returns: this, to enable chaining

setEmptyTokenAsNull

public StrTokenizer setEmptyTokenAsNull(boolean emptyAsNull)
Sets whether the tokenizer should return empty tokens as null. The default for this property is false.

Parameters: emptyAsNull whether empty tokens are returned as null

Returns: this, to enable chaining

setIgnoredChar

public StrTokenizer setIgnoredChar(char ignored)
Set the character to ignore.

This character is ignored when parsing the String, unless it is within a quoted region.

Parameters: ignored the ignored character to use

Returns: this, to enable chaining

setIgnoredMatcher

public StrTokenizer setIgnoredMatcher(StrMatcher ignored)
Set the matcher for characters to ignore.

These characters are ignored when parsing the String, unless they are within a quoted region.

Parameters: ignored the ignored matcher to use, null ignored

Returns: this, to enable chaining

setIgnoreEmptyTokens

public StrTokenizer setIgnoreEmptyTokens(boolean ignoreEmptyTokens)
Sets whether the tokenizer should ignore and not return empty tokens. The default for this property is true.

Parameters: ignoreEmptyTokens whether empty tokens are not returned

Returns: this, to enable chaining

setQuoteChar

public StrTokenizer setQuoteChar(char quote)
Sets the quote character to use.

The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

Parameters: quote the quote character to use

Returns: this, to enable chaining

setQuoteMatcher

public StrTokenizer setQuoteMatcher(StrMatcher quote)
Set the quote matcher to use.

The quote character is used to wrap data between the tokens. This enables delimiters to be entered as data.

Parameters: quote the quote matcher to use, null ignored

Returns: this, to enable chaining

setTrimmerMatcher

public StrTokenizer setTrimmerMatcher(StrMatcher trimmer)
Sets the matcher for characters to trim.

These characters are trimmed off on each side of the delimiter until the token or quote is found.

Parameters: trimmer the trimmer matcher to use, null ignored

Returns: this, to enable chaining

size

public int size()
Gets the number of tokens found in the String.

Returns: the number of matched tokens

tokenize

protected List tokenize(char[] chars, int offset, int count)
Internal method to performs the tokenization.

Most users of this class do not need to call this method. This method will be called automatically by other (public) methods when required.

This method exists to allow subclasses to add code before or after the tokenization. For example, a subclass could alter the character array, offset or count to be parsed, or call the tokenizer multiple times on multiple strings. It is also be possible to filter the results.

StrTokenizer will always pass a zero offset and a count equal to the length of the array to this method, however a subclass may pass other values, or even an entirely different array.

Parameters: chars the character array being tokenized, may be null offset the start position within the character array, must be valid count the number of characters to tokenize, must be valid

Returns: the modifiable list of String tokens, unmodifiable if null array or zero count

toString

public String toString()
Gets the String content that the tokenizer is parsing.

Returns: the string content being parsed

Copyright © 2001-2010 - Apache Software Foundation