|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectweka.core.tokenizers.Tokenizer
weka.core.tokenizers.CharacterDelimitedTokenizer
weka.core.tokenizers.WordTokenizer
public class WordTokenizer
A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.
Valid options are:-delimiters <value> The delimiters to use (default ' \r\n\t.,;:'"()?!').
Constructor Summary | |
---|---|
WordTokenizer()
|
Method Summary | |
---|---|
java.lang.String |
getRevision()
Returns the revision string. |
java.lang.String |
globalInfo()
Returns a string describing the stemmer |
boolean |
hasMoreElements()
Tests if this enumeration contains more elements. |
static void |
main(java.lang.String[] args)
Runs the tokenizer with the given options and strings to tokenize. |
java.lang.Object |
nextElement()
Returns the next element of this enumeration if this enumeration object has at least one more element to provide. |
void |
tokenize(java.lang.String s)
Sets the string to tokenize. |
Methods inherited from class weka.core.tokenizers.CharacterDelimitedTokenizer |
---|
delimitersTipText, getDelimiters, getOptions, listOptions, setDelimiters, setOptions |
Methods inherited from class weka.core.tokenizers.Tokenizer |
---|
runTokenizer, tokenize |
Methods inherited from class java.lang.Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public WordTokenizer()
Method Detail |
---|
public java.lang.String globalInfo()
globalInfo
in class Tokenizer
public boolean hasMoreElements()
hasMoreElements
in interface java.util.Enumeration
hasMoreElements
in class Tokenizer
public java.lang.Object nextElement()
nextElement
in interface java.util.Enumeration
nextElement
in class Tokenizer
public void tokenize(java.lang.String s)
tokenize
in class Tokenizer
s
- the string to tokenizepublic java.lang.String getRevision()
public static void main(java.lang.String[] args)
args
- the commandline options and strings to tokenize
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |