Package org.apache.lucene.codecs.lucene90.blocktree

## Class Lucene90BlockTreeTermsWriter

- java.lang.Object
- - org.apache.lucene.codecs.FieldsConsumer
  - - org.apache.lucene.codecs.lucene90.blocktree.Lucene90BlockTreeTermsWriter

- All Implemented Interfaces:
    `Closeable`, `AutoCloseable`

      public final class Lucene90BlockTreeTermsWriter
      extends FieldsConsumer

  Block-based terms index and dictionary writer.

  Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

  Files:

  - `.tim`: Term Dictionary
  - `.tmd`: Term Metadata
  - `.tip`: Term Index

  ## Term Dictionary

  The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

  The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

  NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

  - TermsDict (.tim) --> Header, FieldDictNumFields, Footer
  - FieldDict --> PostingsHeader, NodeBlockNumBlocks
  - NodeBlock --> (OuterNode | InnerNode)
  - OuterNode --> EntryCount, SuffixLength, ByteSuffixLength, StatsLength, < TermStats >EntryCount, MetaLength, <TermMetadata>EntryCount
  - InnerNode --> EntryCount, SuffixLength[,Sub?], ByteSuffixLength, StatsLength, < TermStats ? >EntryCount, MetaLength, <TermMetadata ? >EntryCount
  - TermStats --> DocFreq, TotalTermFreq
  - Header --> `CodecHeader`
  - EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength --> `VInt`
  - TotalTermFreq --> `VLong`
  - Footer --> `CodecFooter`

  Notes:

  - Header is a `CodecHeader` storing the version information for the BlockTree implementation.
  - DocFreq is the count of documents which contain the term.
  - TotalTermFreq is the total number of occurrences of the term. This is encoded as the difference between the total number of occurrences and the DocFreq.
  - PostingsHeader and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).
  - For inner nodes of the tree, every entry will steal one bit to mark whether it points to child nodes(sub-block). If so, the corresponding TermStats and TermMetaData are omitted.

  ## Term Metadata

  The .tmd file contains the list of term metadata (such as FST index metadata) and field level statistics (such as sum of total term freq).

  - TermsMeta (.tmd) --> Header, NumFields, <FieldStats>NumFields, TermIndexLength, TermDictLength, Footer
  - FieldStats --> FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumTotalTermFreq?, SumDocFreq, DocCount, MinTerm, MaxTerm, IndexStartFP, FSTHeader, FSTMetadata
  - Header,FSTHeader --> `CodecHeader`
  - TermIndexLength, TermDictLength --> `Uint64`
  - MinTerm,MaxTerm --> `VInt` length followed by the byte[]
  - NumFields,FieldNumber,RootCodeLength,DocCount --> `VInt`
  - NumTerms,SumTotalTermFreq,SumDocFreq,IndexStartFP --> `VLong`
  - Footer --> `CodecFooter`

  Notes:

  - FieldNumber is the fields number from `FieldInfos`. (.fnm)
  - NumTerms is the number of unique terms for the field.
  - RootCode points to the root block for the field.
  - SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
  - DocCount is the number of documents that have at least one posting for this field.
  - MinTerm, MaxTerm are the lowest and highest term in this field.

  ## Term Index

  The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

  - TermsIndex (.tip) --> Header, FSTIndexNumFieldsFooter
  - Header --> `CodecHeader`
  - FSTIndex --> `FST<byte[]>`
  - Footer --> `CodecFooter`

  Notes:

  - The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
  - It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.
  See Also:
    `Lucene90BlockTreeTermsReader`
  WARNING: This API is experimental and might change in incompatible ways in the next release.

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static int` | `DEFAULT_MAX_BLOCK_SIZE` | Suggested default value for the `maxItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
    `static int` | `DEFAULT_MIN_BLOCK_SIZE` | Suggested default value for the `minItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `Lucene90BlockTreeTermsWriter​(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock)` | Create a new writer.
    `Lucene90BlockTreeTermsWriter​(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock, int version)` | Expert constructor that allows configuring the version, used for bw tests.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `void` | `close()` |
    `static void` | `validateSettings​(int minItemsInBlock, int maxItemsInBlock)` | Throws `IllegalArgumentException` if any of these settings is invalid.
    `void` | `write​(Fields fields, NormsProducer norms)` | Write all fields, terms and postings.

    - ### Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

      `merge`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_MIN_BLOCK_SIZE

          public static final int DEFAULT_MIN_BLOCK_SIZE

      Suggested default value for the `minItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
      See Also:
        Constant Field Values

    - #### DEFAULT_MAX_BLOCK_SIZE

          public static final int DEFAULT_MAX_BLOCK_SIZE

      Suggested default value for the `maxItemsInBlock` parameter to `Lucene90BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### Lucene90BlockTreeTermsWriter

          public Lucene90BlockTreeTermsWriter​(SegmentWriteState state,
                                              PostingsWriterBase postingsWriter,
                                              int minItemsInBlock,
                                              int maxItemsInBlock)
                                       throws IOException

      Create a new writer. The number of items (terms or sub-blocks) per block will aim to be between minItemsPerBlock and maxItemsPerBlock, though in some cases the blocks may be smaller than the min.
      Throws:
        `IOException`

    - #### Lucene90BlockTreeTermsWriter

          public Lucene90BlockTreeTermsWriter​(SegmentWriteState state,
                                              PostingsWriterBase postingsWriter,
                                              int minItemsInBlock,
                                              int maxItemsInBlock,
                                              int version)
                                       throws IOException

      Expert constructor that allows configuring the version, used for bw tests.
      Throws:
        `IOException`

  - ### Method Detail

    - #### validateSettings

          public static void validateSettings​(int minItemsInBlock,
                                              int maxItemsInBlock)

      Throws `IllegalArgumentException` if any of these settings is invalid.

    - #### write

          public void write​(Fields fields,
                            NormsProducer norms)
                     throws IOException

      Description copied from class: `FieldsConsumer`
      Write all fields, terms and postings. This the "pull" API, allowing you to iterate more than once over the postings, somewhat analogous to using a DOM API to traverse an XML tree.

      Notes:

      - You must compute index statistics, including each Term's docFreq and totalTermFreq, as well as the summary sumTotalTermFreq, sumTotalDocFreq and docCount.
      - You must skip terms that have no docs and fields that have no terms, even though the provided Fields API will expose them; this typically requires lazily writing the field or term until you've actually seen the first term or document.
      - The provided Fields instance is limited: you cannot call any methods that return statistics/counts; you cannot pass a non-null live docs when pulling docs/positions enums.
      Specified by:
        `write` in class `FieldsConsumer`
      Throws:
        `IOException`

    - #### close

          public void close()
                     throws IOException

      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in class `FieldsConsumer`
      Throws:
        `IOException`