A grammar-based tokenizer constructed with JFlex

This should be a good tokenizer for most European-language documents:

  • Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
  • Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
  • Recognizes email addresses and internet hostnames as one token.

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer. ClassicTokenizer was named StandardTokenizer in Lucene versions prior to 3.1. As of 3.1, StandardTokenizer implements Unicode text segmentation, as specified by UAX#29.

Inheritance: Tokenizer
ClassicTokenizer Class Documentation
Esempio n. 1
0
        public override ClassicTokenizer create(AttributeFactory factory, Reader input)
        {
            ClassicTokenizer tokenizer = new ClassicTokenizer(luceneMatchVersion, factory, input);

            tokenizer.MaxTokenLength = maxTokenLength;
            return(tokenizer);
        }
Esempio n. 2
0
//JAVA TO C# CONVERTER WARNING: 'final' parameters are not available in .NET:
//ORIGINAL LINE: @Override protected TokenStreamComponents createComponents(final String fieldName, final java.io.Reader reader)
        protected internal override TokenStreamComponents createComponents(string fieldName, Reader reader)
        {
//JAVA TO C# CONVERTER WARNING: The original Java variable was marked 'final':
//ORIGINAL LINE: final ClassicTokenizer src = new ClassicTokenizer(matchVersion, reader);
            ClassicTokenizer src = new ClassicTokenizer(matchVersion, reader);

            src.MaxTokenLength = maxTokenLength;
            TokenStream tok = new ClassicFilter(src);

            tok = new LowerCaseFilter(matchVersion, tok);
            tok = new StopFilter(matchVersion, tok, stopwords);
            return(new TokenStreamComponentsAnonymousInnerClassHelper(this, src, tok, reader));
        }