## Uses of Class
org.apache.lucene.analysis.Tokenizer

- Packages that use Tokenizer
  Package | Description
  --- | ---
  org.apache.lucene.analysis | Text analysis.
  org.apache.lucene.analysis.standard | Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.

- - ### Uses of Tokenizer in org.apache.lucene.analysis

    Methods in org.apache.lucene.analysis that return Tokenizer
    Modifier and Type | Method | Description
    --- | --- | ---
    `Tokenizer` | TokenizerFactory.`create()` | Creates a TokenStream of the specified input using the default attribute factory.
    `abstract Tokenizer` | TokenizerFactory.`create​(AttributeFactory factory)` | Creates a TokenStream of the specified input using the given AttributeFactory

    Constructors in org.apache.lucene.analysis with parameters of type Tokenizer
    Constructor | Description
    --- | ---
    `TokenStreamComponents​(Tokenizer tokenizer)` | Creates a new `Analyzer.TokenStreamComponents` from a Tokenizer
    `TokenStreamComponents​(Tokenizer tokenizer, TokenStream result)` | Creates a new `Analyzer.TokenStreamComponents` instance

  - ### Uses of Tokenizer in org.apache.lucene.analysis.standard

    Subclasses of Tokenizer in org.apache.lucene.analysis.standard
    Modifier and Type | Class | Description
    --- | --- | ---
    `class` | `StandardTokenizer` | A grammar-based tokenizer constructed with JFlex.