## Uses of Class
org.apache.lucene.analysis.TokenStream

- Packages that use TokenStream
  Package | Description
  --- | ---
  org.apache.lucene.analysis | Text analysis.
  org.apache.lucene.analysis.standard | Fast, general-purpose grammar-based tokenizer `StandardTokenizer` implements the Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29.
  org.apache.lucene.document | The logical representation of a `Document` for indexing and searching.
  org.apache.lucene.index | Code to maintain and access indices.
  org.apache.lucene.util | Some utility classes.
  org.apache.lucene.util.graph | Utility classes for working with token streams as graphs.

- - ### Uses of TokenStream in org.apache.lucene.analysis

    Subclasses of TokenStream in org.apache.lucene.analysis
    Modifier and Type | Class | Description
    --- | --- | ---
    `class` | `CachingTokenFilter` | This class can be used if the token attributes of a TokenStream are intended to be consumed more than once.
    `class` | `FilteringTokenFilter` | Abstract base class for TokenFilters that may remove tokens.
    `class` | `GraphTokenFilter` | An abstract TokenFilter that exposes its input stream as a graph
    `class` | `LowerCaseFilter` | Normalizes token text to lower case.
    `class` | `StopFilter` | Removes stop words from a token stream.
    `class` | `TokenFilter` | A TokenFilter is a TokenStream whose input is another TokenStream.
    `class` | `Tokenizer` | A Tokenizer is a TokenStream whose input is a Reader.

    Fields in org.apache.lucene.analysis declared as TokenStream
    Modifier and Type | Field | Description
    --- | --- | ---
    `protected TokenStream` | TokenFilter.`input` | The source of tokens for this filter.
    `protected TokenStream` | Analyzer.TokenStreamComponents.`sink` | Sink tokenstream, such as the outer tokenfilter decorating the chain.

    Methods in org.apache.lucene.analysis that return TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `abstract TokenStream` | TokenFilterFactory.`create​(TokenStream input)` | Transform the specified input TokenStream
    `TokenStream` | Analyzer.TokenStreamComponents.`getTokenStream()` | Returns the sink `TokenStream`
    `protected TokenStream` | Analyzer.`normalize​(String fieldName, TokenStream in)` | Wrap the given `TokenStream` in order to apply normalization filters.
    `protected TokenStream` | AnalyzerWrapper.`normalize​(String fieldName, TokenStream in)` |
    `TokenStream` | TokenFilterFactory.`normalize​(TokenStream input)` | Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to `create` method.
    `TokenStream` | Analyzer.`tokenStream​(String fieldName, Reader reader)` | Returns a TokenStream suitable for `fieldName`, tokenizing the contents of `reader`.
    `TokenStream` | Analyzer.`tokenStream​(String fieldName, String text)` | Returns a TokenStream suitable for `fieldName`, tokenizing the contents of `text`.
    `static TokenStream` | AutomatonToTokenStream.`toTokenStream​(Automaton automaton)` | converts an automaton into a TokenStream.
    `TokenStream` | TokenFilter.`unwrap()` |
    `protected TokenStream` | AnalyzerWrapper.`wrapTokenStreamForNormalization​(String fieldName, TokenStream in)` | Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.
    `protected TokenStream` | DelegatingAnalyzerWrapper.`wrapTokenStreamForNormalization​(String fieldName, TokenStream in)` |

    Methods in org.apache.lucene.analysis with parameters of type TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `abstract TokenStream` | TokenFilterFactory.`create​(TokenStream input)` | Transform the specified input TokenStream
    `protected TokenStream` | Analyzer.`normalize​(String fieldName, TokenStream in)` | Wrap the given `TokenStream` in order to apply normalization filters.
    `protected TokenStream` | AnalyzerWrapper.`normalize​(String fieldName, TokenStream in)` |
    `TokenStream` | TokenFilterFactory.`normalize​(TokenStream input)` | Normalize the specified input TokenStream While the default implementation returns input unchanged, filters that should be applied at normalization time can delegate to `create` method.
    `Automaton` | TokenStreamToAutomaton.`toAutomaton​(TokenStream in)` | Pulls the graph (including `PositionLengthAttribute`) from the provided `TokenStream`, and creates the corresponding automaton where arcs are bytes (or Unicode code points if unicodeArcs = true) from each term.
    `protected TokenStream` | AnalyzerWrapper.`wrapTokenStreamForNormalization​(String fieldName, TokenStream in)` | Wraps / alters the given TokenStream for normalization purposes, taken from the wrapped Analyzer, to form new components.
    `protected TokenStream` | DelegatingAnalyzerWrapper.`wrapTokenStreamForNormalization​(String fieldName, TokenStream in)` |

    Constructors in org.apache.lucene.analysis with parameters of type TokenStream
    Constructor | Description
    --- | ---
    `CachingTokenFilter​(TokenStream input)` | Create a new CachingTokenFilter around `input`.
    `FilteringTokenFilter​(TokenStream in)` | Create a new `FilteringTokenFilter`.
    `GraphTokenFilter​(TokenStream input)` | Create a new GraphTokenFilter
    `LowerCaseFilter​(TokenStream in)` | Create a new LowerCaseFilter, that normalizes token text to lower case.
    `StopFilter​(TokenStream in, CharArraySet stopWords)` | Constructs a filter which removes words from the input TokenStream that are named in the Set.
    `TokenFilter​(TokenStream input)` | Construct a token stream filtering the given input.
    `TokenStreamComponents​(Consumer<Reader> source, TokenStream result)` | Creates a new `Analyzer.TokenStreamComponents` instance.
    `TokenStreamComponents​(Tokenizer tokenizer, TokenStream result)` | Creates a new `Analyzer.TokenStreamComponents` instance

  - ### Uses of TokenStream in org.apache.lucene.analysis.standard

    Subclasses of TokenStream in org.apache.lucene.analysis.standard
    Modifier and Type | Class | Description
    --- | --- | ---
    `class` | `StandardTokenizer` | A grammar-based tokenizer constructed with JFlex.

    Methods in org.apache.lucene.analysis.standard that return TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `protected TokenStream` | StandardAnalyzer.`normalize​(String fieldName, TokenStream in)` |

    Methods in org.apache.lucene.analysis.standard with parameters of type TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `protected TokenStream` | StandardAnalyzer.`normalize​(String fieldName, TokenStream in)` |

  - ### Uses of TokenStream in org.apache.lucene.document

    Fields in org.apache.lucene.document declared as TokenStream
    Modifier and Type | Field | Description
    --- | --- | ---
    `protected TokenStream` | Field.`tokenStream` | Pre-analyzed tokenStream for indexed fields; this is separate from fieldsData because you are allowed to have both; eg maybe field has a String value but you customize how it's tokenized

    Methods in org.apache.lucene.document that return TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `TokenStream` | FeatureField.`tokenStream​(Analyzer analyzer, TokenStream reuse)` |
    `TokenStream` | Field.`tokenStream​(Analyzer analyzer, TokenStream reuse)` |
    `TokenStream` | ShapeDocValuesField.`tokenStream​(Analyzer analyzer, TokenStream reuse)` | TokenStreams are not yet supported
    `TokenStream` | Field.`tokenStreamValue()` | The TokenStream for this field to be used when indexing, or null.

    Methods in org.apache.lucene.document with parameters of type TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `void` | Field.`setTokenStream​(TokenStream tokenStream)` | Expert: sets the token stream to be used for indexing and causes isIndexed() and isTokenized() to return true.
    `TokenStream` | FeatureField.`tokenStream​(Analyzer analyzer, TokenStream reuse)` |
    `TokenStream` | Field.`tokenStream​(Analyzer analyzer, TokenStream reuse)` |
    `TokenStream` | ShapeDocValuesField.`tokenStream​(Analyzer analyzer, TokenStream reuse)` | TokenStreams are not yet supported

    Constructors in org.apache.lucene.document with parameters of type TokenStream
    Constructor | Description
    --- | ---
    `Field​(String name, TokenStream tokenStream, IndexableFieldType type)` | Create field with TokenStream value.
    `TextField​(String name, TokenStream stream)` | Creates a new un-stored TextField with TokenStream value.

  - ### Uses of TokenStream in org.apache.lucene.index

    Methods in org.apache.lucene.index that return TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `TokenStream` | IndexableField.`tokenStream​(Analyzer analyzer, TokenStream reuse)` | Creates the TokenStream used for indexing this field.

    Methods in org.apache.lucene.index with parameters of type TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `TokenStream` | IndexableField.`tokenStream​(Analyzer analyzer, TokenStream reuse)` | Creates the TokenStream used for indexing this field.

  - ### Uses of TokenStream in org.apache.lucene.util

    Methods in org.apache.lucene.util with parameters of type TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `protected Query` | QueryBuilder.`analyzeBoolean​(String field, TokenStream stream)` | Creates simple boolean query from the cached tokenstream contents
    `protected Query` | QueryBuilder.`analyzeGraphBoolean​(String field, TokenStream source, BooleanClause.Occur operator)` | Creates a boolean query from a graph token stream.
    `protected Query` | QueryBuilder.`analyzeGraphPhrase​(TokenStream source, String field, int phraseSlop)` | Creates graph phrase query from the tokenstream contents
    `protected Query` | QueryBuilder.`analyzeMultiBoolean​(String field, TokenStream stream, BooleanClause.Occur operator)` | Creates complex boolean query from the cached tokenstream contents
    `protected Query` | QueryBuilder.`analyzeMultiPhrase​(String field, TokenStream stream, int slop)` | Creates complex phrase query from the cached tokenstream contents
    `protected Query` | QueryBuilder.`analyzePhrase​(String field, TokenStream stream, int slop)` | Creates simple phrase query from the cached tokenstream contents
    `protected Query` | QueryBuilder.`analyzeTerm​(String field, TokenStream stream)` | Creates simple term query from the cached tokenstream contents
    `protected Query` | QueryBuilder.`createFieldQuery​(TokenStream source, BooleanClause.Occur operator, String field, boolean quoted, int phraseSlop)` | Creates a query from a token stream.

  - ### Uses of TokenStream in org.apache.lucene.util.graph

    Methods in org.apache.lucene.util.graph that return types with arguments of type TokenStream
    Modifier and Type | Method | Description
    --- | --- | ---
    `Iterator<TokenStream>` | GraphTokenStreamFiniteStrings.`getFiniteStrings()` | Get all finite strings from the automaton.
    `Iterator<TokenStream>` | GraphTokenStreamFiniteStrings.`getFiniteStrings​(int startState, int endState)` | Get all finite strings that start at `startState` and end at `endState`.

    Constructors in org.apache.lucene.util.graph with parameters of type TokenStream
    Constructor | Description
    --- | ---
    `GraphTokenStreamFiniteStrings​(TokenStream in)` |