Class | Ferret::Analysis::AsciiStandardTokenizer |
In: |
ext/r_analysis.c
|
Parent: | Ferret::Analysis::TokenStream |
The standard tokenizer is an advanced tokenizer which tokenizes most words correctly as well as tokenizing things like email addresses, web addresses, phone numbers, etc.
"Dave's résumé, at http://www.davebalmain.com/ 1234" => ["Dave's", "r", "sum", "at", "http://www.davebalmain.com", "1234"]