Uses of Class
cc.mallet.pipe.SimpleTokenizer

Packages that use SimpleTokenizer

Package	Description
cc.mallet.pipe	Classes for processing arbitrary data into instances.
cc.mallet.util	Miscellaneous utilities including command line processing, math functions, lexing, logging.

Uses of SimpleTokenizer in cc.mallet.pipe

Methods in cc.mallet.pipe that return SimpleTokenizer

Modifier and Type	Method	Description
SimpleTokenizer	SimpleTokenizer.deepClone()

Methods in cc.mallet.pipe with parameters of type SimpleTokenizer

Modifier and Type	Method	Description
void	FeatureCountPipe.addPrunedWordsToStoplist(SimpleTokenizer tokenizer, int minimumCount)	Add all pruned words to the internal stoplist of a SimpleTokenizer.
void	FeatureDocFreqPipe.addPrunedWordsToStoplist(SimpleTokenizer tokenizer, double docFrequencyCutoff)	Add all pruned words to the internal stoplist of a SimpleTokenizer.

Uses of SimpleTokenizer in cc.mallet.util

Methods in cc.mallet.util with parameters of type SimpleTokenizer

Modifier and Type	Method	Description
static void	BulkLoader.generateStoplist(SimpleTokenizer prunedTokenizer)	Read the data from inputFile, then write all the words that do not occur pruneCount.value times or more to the pruned word file.
static void	DBBulkLoader.generateStoplist(SimpleTokenizer prunedTokenizer, NGramPreprocessor preprocessor)	Read the data from inputFiles, then write all the words that do not occur pruneCount.value times or more to the pruned word file.
static void	BulkLoader.writeInstanceList(SimpleTokenizer prunedTokenizer)