Package org.apache.lucene.index

## Class CheckIndex

- java.lang.Object
- - org.apache.lucene.index.CheckIndex

- All Implemented Interfaces:
    `Closeable`, `AutoCloseable`

      public final class CheckIndex
      extends Object
      implements Closeable

  Basic tool and API to check the health of an index and write a new segments file that removes reference to problematic segments.

  As this tool checks every byte in the index, on a large index it can take quite a long time to run.

  WARNING: This API is experimental and might change in incompatible ways in the next release.
    Please make a complete backup of your index before using this to exorcise corrupted documents from your index!

- - ### Nested Class Summary

    Nested Classes
    Modifier and Type | Class | Description
    --- | --- | ---
    `static class` | `CheckIndex.CheckIndexException` | The marker RuntimeException used by CheckIndex APIs when index integrity failure is detected.
    `static class` | `CheckIndex.Options` | Run-time configuration options for CheckIndex commands.
    `static class` | `CheckIndex.Status` | Returned from `checkIndex()` detailing the health and status of the index.
    `static class` | `CheckIndex.VerifyPointsVisitor` | Walks the entire N-dimensional points space, verifying that all points fall within the last cell's boundaries.

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `CheckIndex​(Directory dir)` | Create a new CheckIndex on the directory.
    `CheckIndex​(Directory dir, Lock writeLock)` | Expert: create a directory with the specified lock.

  - ### Method Summary

    All Methods Static Methods Instance Methods Concrete Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `static boolean` | `assertsOn()` | Check whether asserts are enabled or not.
    `CheckIndex.Status` | `checkIndex()` | Returns a `CheckIndex.Status` instance detailing the state of the index.
    `CheckIndex.Status` | `checkIndex​(List<String> onlySegments)` | Returns a `CheckIndex.Status` instance detailing the state of the index.
    `CheckIndex.Status` | `checkIndex​(List<String> onlySegments, ExecutorService executorService)` | Returns a `CheckIndex.Status` instance detailing the state of the index.
    `void` | `close()` |
    `int` | `doCheck​(CheckIndex.Options opts)` | Actually perform the index check
    `boolean` | `doSlowChecks()` | See `setDoSlowChecks(boolean)`.
    `void` | `exorciseIndex​(CheckIndex.Status result)` | Repairs the index using previously returned result from `checkIndex()`.
    `boolean` | `getChecksumsOnly()` | See `getChecksumsOnly()`.
    `boolean` | `getFailFast()` | See `setFailFast(boolean)`.
    `static void` | `main​(String[] args)` | Command-line interface to check and exorcise corrupt segments from an index.
    `static CheckIndex.Options` | `parseOptions​(String[] args)` | Parse command line args into fields
    `void` | `setChecksumsOnly​(boolean v)` | If true, only validate physical integrity for all files.
    `void` | `setDoSlowChecks​(boolean v)` | If true, additional slow checks are performed.
    `void` | `setFailFast​(boolean v)` | If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.
    `void` | `setInfoStream​(PrintStream out)` | Set infoStream where messages should go.
    `void` | `setInfoStream​(PrintStream out, boolean verbose)` | Set infoStream where messages should go.
    `void` | `setThreadCount​(int tc)` | Set threadCount used for parallelizing index integrity checking.
    `static CheckIndex.Status.DocValuesStatus` | `testDocValues​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test docvalues.
    `static CheckIndex.Status.FieldInfoStatus` | `testFieldInfos​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test field infos.
    `static CheckIndex.Status.FieldNormStatus` | `testFieldNorms​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test field norms.
    `static CheckIndex.Status.LiveDocStatus` | `testLiveDocs​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test live docs.
    `static CheckIndex.Status.PointsStatus` | `testPoints​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test the points index.
    `static CheckIndex.Status.TermIndexStatus` | `testPostings​(CodecReader reader, PrintStream infoStream)` | Test the term index.
    `static CheckIndex.Status.TermIndexStatus` | `testPostings​(CodecReader reader, PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)` | Test the term index.
    `static CheckIndex.Status.IndexSortStatus` | `testSort​(CodecReader reader, Sort sort, PrintStream infoStream, boolean failFast)` | Tests index sort order.
    `static CheckIndex.Status.StoredFieldStatus` | `testStoredFields​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test stored fields.
    `static CheckIndex.Status.TermVectorStatus` | `testTermVectors​(CodecReader reader, PrintStream infoStream)` | Test term vectors.
    `static CheckIndex.Status.TermVectorStatus` | `testTermVectors​(CodecReader reader, PrintStream infoStream, boolean verbose, boolean doSlowChecks, boolean failFast)` | Test term vectors.
    `static CheckIndex.Status.VectorValuesStatus` | `testVectors​(CodecReader reader, PrintStream infoStream, boolean failFast)` | Test the vectors index.

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Constructor Detail

    - #### CheckIndex

          public CheckIndex​(Directory dir)
                     throws IOException

      Create a new CheckIndex on the directory.
      Throws:
        `IOException`

    - #### CheckIndex

          public CheckIndex​(Directory dir,
                            Lock writeLock)

      Expert: create a directory with the specified lock. This should really not be used except for unit tests!!!! It exists only to support special tests (such as TestIndexWriterExceptions*), that would otherwise be more complicated to debug if they had to close the writer for each check.

  - ### Method Detail

    - #### close

          public void close()
                     throws IOException

      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in interface `Closeable`
      Throws:
        `IOException`

    - #### setDoSlowChecks

          public void setDoSlowChecks​(boolean v)

      If true, additional slow checks are performed. This will likely drastically increase time it takes to run CheckIndex!

    - #### doSlowChecks

          public boolean doSlowChecks()

      See `setDoSlowChecks(boolean)`.

    - #### setFailFast

          public void setFailFast​(boolean v)

      If true, just throw the original exception immediately when corruption is detected, rather than continuing to iterate to other segments looking for more corruption.

    - #### getFailFast

          public boolean getFailFast()

      See `setFailFast(boolean)`.

    - #### getChecksumsOnly

          public boolean getChecksumsOnly()

      See `getChecksumsOnly()`.

    - #### setChecksumsOnly

          public void setChecksumsOnly​(boolean v)

      If true, only validate physical integrity for all files. Note that the returned nested status objects (e.g. storedFieldStatus) will be null.

    - #### setThreadCount

          public void setThreadCount​(int tc)

      Set threadCount used for parallelizing index integrity checking.

    - #### setInfoStream

          public void setInfoStream​(PrintStream out,
                                    boolean verbose)

      Set infoStream where messages should go. If null, no messages are printed. If verbose is true then more details are printed.

    - #### setInfoStream

          public void setInfoStream​(PrintStream out)

      Set infoStream where messages should go. See `setInfoStream(PrintStream,boolean)`.

    - #### checkIndex

          public CheckIndex.Status checkIndex()
                                       throws IOException

      Returns a `CheckIndex.Status` instance detailing the state of the index.

      As this method checks every byte in the index, on a large index it can take quite a long time to run.

      WARNING: make sure you only call this when the index is not opened by any writer.

      Throws:
        `IOException`

    - #### checkIndex

          public CheckIndex.Status checkIndex​(List<String> onlySegments)
                                       throws IOException

      Returns a `CheckIndex.Status` instance detailing the state of the index.
      Parameters:
        `onlySegments` - list of specific segment names to check

        As this method checks every byte in the specified segments, on a large index it can take quite a long time to run.

      Throws:
        `IOException`

    - #### checkIndex

          public CheckIndex.Status checkIndex​(List<String> onlySegments,
                                              ExecutorService executorService)
                                       throws IOException

      Returns a `CheckIndex.Status` instance detailing the state of the index.

      This method allows caller to pass in customized ExecutorService to speed up the check.

      WARNING: make sure you only call this when the index is not opened by any writer.

      Throws:
        `IOException`

    - #### testSort

          public static CheckIndex.Status.IndexSortStatus testSort​(CodecReader reader,
                                                                   Sort sort,
                                                                   PrintStream infoStream,
                                                                   boolean failFast)
                                                            throws IOException

      Tests index sort order.
      Throws:
        `IOException`

    - #### testLiveDocs

          public static CheckIndex.Status.LiveDocStatus testLiveDocs​(CodecReader reader,
                                                                     PrintStream infoStream,
                                                                     boolean failFast)
                                                              throws IOException

      Test live docs.
      Throws:
        `IOException`

    - #### testFieldInfos

          public static CheckIndex.Status.FieldInfoStatus testFieldInfos​(CodecReader reader,
                                                                         PrintStream infoStream,
                                                                         boolean failFast)
                                                                  throws IOException

      Test field infos.
      Throws:
        `IOException`

    - #### testFieldNorms

          public static CheckIndex.Status.FieldNormStatus testFieldNorms​(CodecReader reader,
                                                                         PrintStream infoStream,
                                                                         boolean failFast)
                                                                  throws IOException

      Test field norms.
      Throws:
        `IOException`

    - #### testPostings

          public static CheckIndex.Status.TermIndexStatus testPostings​(CodecReader reader,
                                                                       PrintStream infoStream)
                                                                throws IOException

      Test the term index.
      Throws:
        `IOException`

    - #### testPostings

          public static CheckIndex.Status.TermIndexStatus testPostings​(CodecReader reader,
                                                                       PrintStream infoStream,
                                                                       boolean verbose,
                                                                       boolean doSlowChecks,
                                                                       boolean failFast)
                                                                throws IOException

      Test the term index.
      Throws:
        `IOException`

    - #### testPoints

          public static CheckIndex.Status.PointsStatus testPoints​(CodecReader reader,
                                                                  PrintStream infoStream,
                                                                  boolean failFast)
                                                           throws IOException

      Test the points index.
      Throws:
        `IOException`

    - #### testVectors

          public static CheckIndex.Status.VectorValuesStatus testVectors​(CodecReader reader,
                                                                         PrintStream infoStream,
                                                                         boolean failFast)
                                                                  throws IOException

      Test the vectors index.
      Throws:
        `IOException`

    - #### testStoredFields

          public static CheckIndex.Status.StoredFieldStatus testStoredFields​(CodecReader reader,
                                                                             PrintStream infoStream,
                                                                             boolean failFast)
                                                                      throws IOException

      Test stored fields.
      Throws:
        `IOException`

    - #### testDocValues

          public static CheckIndex.Status.DocValuesStatus testDocValues​(CodecReader reader,
                                                                        PrintStream infoStream,
                                                                        boolean failFast)
                                                                 throws IOException

      Test docvalues.
      Throws:
        `IOException`

    - #### testTermVectors

          public static CheckIndex.Status.TermVectorStatus testTermVectors​(CodecReader reader,
                                                                           PrintStream infoStream)
                                                                    throws IOException

      Test term vectors.
      Throws:
        `IOException`

    - #### testTermVectors

          public static CheckIndex.Status.TermVectorStatus testTermVectors​(CodecReader reader,
                                                                           PrintStream infoStream,
                                                                           boolean verbose,
                                                                           boolean doSlowChecks,
                                                                           boolean failFast)
                                                                    throws IOException

      Test term vectors.
      Throws:
        `IOException`

    - #### exorciseIndex

          public void exorciseIndex​(CheckIndex.Status result)
                             throws IOException

      Repairs the index using previously returned result from `checkIndex()`. Note that this does not remove any of the unreferenced files after it's done; you must separately open an `IndexWriter`, which deletes unreferenced files when it's created.

      WARNING: this writes a new segments file into the index, effectively removing all documents in broken segments from the index. BE CAREFUL.

      Throws:
        `IOException`

    - #### assertsOn

          public static boolean assertsOn()

      Check whether asserts are enabled or not.
      Returns:
        true iff asserts are enabled

    - #### main

          public static void main​(String[] args)
                           throws IOException,
                                  InterruptedException

      Command-line interface to check and exorcise corrupt segments from an index.

      Run it like this:

           java -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex pathToIndex [-exorcise] [-verbose] [-segment X] [-segment Y]

      - `-exorcise`: actually write a new segments_N file, removing any problematic segments. *LOSES DATA*
      - `-segment X`: only check the specified segment(s). This can be specified multiple times, to check more than one segment: `-segment _2 * -segment _a`. You can't use this with the -exorcise option.

      WARNING: `-exorcise` should only be used on an emergency basis as it will cause documents (perhaps many) to be permanently removed from the index. Always make a backup copy of your index before running this! Do not run this tool on an index that is actively being written to. You have been warned!

      Run without -exorcise, this tool will open the index, report version information and report any exceptions it hits and what action it would take if -exorcise were specified. With -exorcise, this tool will remove any segments that have issues and write a new segments_N file. This means all documents contained in the affected segments will be removed.

      This tool exits with exit code 1 if the index cannot be opened or has any corruption, else 0.

      Throws:
        `IOException`
        `InterruptedException`

    - #### parseOptions

          public static CheckIndex.Options parseOptions​(String[] args)

      Parse command line args into fields
      Parameters:
        `args` - The command line arguments
      Returns:
        An Options struct
      Throws:
        `IllegalArgumentException` - if any of the CLI args are invalid

    - #### doCheck

          public int doCheck​(CheckIndex.Options opts)
                      throws IOException,
                             InterruptedException

      Actually perform the index check
      Parameters:
        `opts` - The options to use for this check
      Returns:
        0 iff the index is clean, 1 otherwise
      Throws:
        `IOException`
        `InterruptedException`