Package org.apache.lucene.store

## Class MMapDirectory

- java.lang.Object
- - org.apache.lucene.store.Directory
  - - org.apache.lucene.store.BaseDirectory
    - - org.apache.lucene.store.FSDirectory
      - - org.apache.lucene.store.MMapDirectory

- All Implemented Interfaces:
    `Closeable`, `AutoCloseable`

      public class MMapDirectory
      extends FSDirectory

  File-based `Directory` implementation that uses mmap for reading, and `FSDirectory.FSIndexOutput` for writing.

  NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space. On 32 bit platforms also consult `MMapDirectory(Path, LockFactory, long)` if you have problems with mmap failing because of fragmented address space. If you get an OutOfMemoryException, it is recommended to reduce the chunk size, until it works.

  This class supports preloading files into physical memory upon opening. This can help improve performance of searches on a cold page cache at the expense of slowing down opening an index. See `setPreload(BiPredicate)` for more details.

  Due to this bug in OpenJDK, MMapDirectory's `IndexInput.close()` is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed.

  This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of.

  This class supplies the workaround mentioned in the bug report, which may fail on non-Oracle/OpenJDK JVMs. It forcefully unmaps the buffer on close by using an undocumented internal cleanup functionality. If `UNMAP_SUPPORTED` is `true`, the workaround will be automatically enabled (with no guarantees; if you discover any problems, you can disable it by using system property `ENABLE_UNMAP_HACK_SYSPROP`).

  For the hack to work correct, the following requirements need to be fulfilled: The used JVM must be at least Oracle Java / OpenJDK. In addition, the following permissions need to be granted to `lucene-core.jar` in your policy file:

  - `permission java.lang.reflect.ReflectPermission "suppressAccessChecks";`
  - `permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";`

  On exactly Java 19 / 20 / 21 this class will use the modern `MemorySegment` API which allows to safely unmap (if you discover any problems with this preview API, you can disable it by using system property `ENABLE_MEMORY_SEGMENTS_SYSPROP`).

  NOTE: Accessing this class either directly or indirectly from a thread while it's interrupted can close the underlying channel immediately if at the same time the thread is blocked on IO. The channel will remain closed and subsequent access to `MMapDirectory` will throw a `ClosedChannelException`. If your application uses either `Thread.interrupt()` or `Future.cancel(boolean)` you should use the legacy `RAFDirectory` from the Lucene `misc` module in favor of `MMapDirectory`.

  NOTE: If your application requires external synchronization, you should not synchronize on the `MMapDirectory` instance as this may cause deadlock; use your own (non-Lucene) objects instead.

  See Also:
    Blog post about MMapDirectory

- - ### Field Summary

    Fields
    Modifier and Type | Field | Description
    --- | --- | ---
    `static BiPredicate<String,​IOContext>` | `ALL_FILES` | Argument for `setPreload(BiPredicate)` that configures all files to be preloaded upon opening them.
    `static BiPredicate<String,​IOContext>` | `BASED_ON_LOAD_IO_CONTEXT` | Argument for `setPreload(BiPredicate)` that configures files to be preloaded upon opening them if they use the `IOContext.LOAD` I/O context.
    `static long` | `DEFAULT_MAX_CHUNK_SIZE` | Default max chunk size: 16 GiBytes for 64 bit Java 19 / 20 / 21 JVMs 1 GiBytes for other 64 bit JVMs 256 MiBytes for 32 bit JVMs
    `static String` | `ENABLE_MEMORY_SEGMENTS_SYSPROP` | This sysprop allows to control if `MemorySegment` API should be used on supported Java versions.
    `static String` | `ENABLE_UNMAP_HACK_SYSPROP` | This sysprop allows to control the workaround/hack for unmapping the buffers from address space after closing `IndexInput`.
    `static BiPredicate<String,​IOContext>` | `NO_FILES` | Argument for `setPreload(BiPredicate)` that configures no files to be preloaded upon opening them.
    `static String` | `UNMAP_NOT_SUPPORTED_REASON` | if `UNMAP_SUPPORTED` is `false`, this contains the reason why unmapping is not supported.
    `static boolean` | `UNMAP_SUPPORTED` | `true`, if this platform supports unmapping mmapped files.

    - ### Fields inherited from class org.apache.lucene.store.FSDirectory

      `directory`

    - ### Fields inherited from class org.apache.lucene.store.BaseDirectory

      `isOpen, lockFactory`

  - ### Constructor Summary

    Constructors
    Constructor | Description
    --- | ---
    `MMapDirectory​(Path path)` | Create a new MMapDirectory for the named location and `FSLockFactory.getDefault()`.
    `MMapDirectory​(Path path, int maxChunkSize)` | Deprecated. use `MMapDirectory(Path, long)` instead.
    `MMapDirectory​(Path path, long maxChunkSize)` | Create a new MMapDirectory for the named location and `FSLockFactory.getDefault()`.
    `MMapDirectory​(Path path, LockFactory lockFactory)` | Create a new MMapDirectory for the named location.
    `MMapDirectory​(Path path, LockFactory lockFactory, int maxChunkSize)` | Deprecated. use `MMapDirectory(Path, LockFactory, long)` instead.
    `MMapDirectory​(Path path, LockFactory lockFactory, long maxChunkSize)` | Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping.

  - ### Method Summary

    All Methods Instance Methods Concrete Methods Deprecated Methods
    Modifier and Type | Method | Description
    --- | --- | ---
    `long` | `getMaxChunkSize()` | Returns the current mmap chunk size.
    `boolean` | `getPreload()` | Deprecated. This information is no longer reliable now that preloading is more granularly configured via a predicate.
    `boolean` | `getUseUnmap()` | Deprecated. use `UNMAP_SUPPORTED`
    `IndexInput` | `openInput​(String name, IOContext context)` | Creates an IndexInput for the file with the given name.
    `void` | `setPreload​(boolean preload)` | Deprecated. Use `setPreload(BiPredicate)` instead which provides more granular control.
    `void` | `setPreload​(BiPredicate<String,​IOContext> preload)` | Configure which files to preload in physical memory upon opening.
    `void` | `setUseUnmap​(boolean useUnmapHack)` | Deprecated, for removal: This API element is subject to removal in a future version. Please use new system property `ENABLE_UNMAP_HACK_SYSPROP` instead

    - ### Methods inherited from class org.apache.lucene.store.FSDirectory

      `close, createOutput, createTempOutput, deleteFile, deletePendingFiles, ensureCanRead, fileLength, fsync, getDirectory, getPendingDeletions, listAll, listAll, open, open, rename, sync, syncMetaData, toString`

    - ### Methods inherited from class org.apache.lucene.store.BaseDirectory

      `ensureOpen, obtainLock`

    - ### Methods inherited from class org.apache.lucene.store.Directory

      `copyFrom, getTempFileName, openChecksumInput`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- - ### Field Detail

    - #### ALL_FILES

          public static final BiPredicate<String,​IOContext> ALL_FILES

      Argument for `setPreload(BiPredicate)` that configures all files to be preloaded upon opening them.

    - #### NO_FILES

          public static final BiPredicate<String,​IOContext> NO_FILES

      Argument for `setPreload(BiPredicate)` that configures no files to be preloaded upon opening them.

    - #### BASED_ON_LOAD_IO_CONTEXT

          public static final BiPredicate<String,​IOContext> BASED_ON_LOAD_IO_CONTEXT

      Argument for `setPreload(BiPredicate)` that configures files to be preloaded upon opening them if they use the `IOContext.LOAD` I/O context.

    - #### DEFAULT_MAX_CHUNK_SIZE

          public static final long DEFAULT_MAX_CHUNK_SIZE

      Default max chunk size:
      - 16 GiBytes for 64 bit Java 19 / 20 / 21 JVMs
      - 1 GiBytes for other 64 bit JVMs
      - 256 MiBytes for 32 bit JVMs

    - #### ENABLE_UNMAP_HACK_SYSPROP

          public static final String ENABLE_UNMAP_HACK_SYSPROP

      This sysprop allows to control the workaround/hack for unmapping the buffers from address space after closing `IndexInput`. By default it is enabled; set to `false` to disable the unmap hack globally. On command line pass `-Dorg.apache.lucene.store.MMapDirectory.enableUnmapHack=false` to disable.
      See Also:
        Constant Field Values
      NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

    - #### ENABLE_MEMORY_SEGMENTS_SYSPROP

          public static final String ENABLE_MEMORY_SEGMENTS_SYSPROP

      This sysprop allows to control if `MemorySegment` API should be used on supported Java versions. By default it is enabled; set to `false` to use legacy `ByteBuffer` implementation. On command line pass `-Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false` to disable.
      See Also:
        Constant Field Values
      NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.

    - #### UNMAP_SUPPORTED

          public static final boolean UNMAP_SUPPORTED

      `true`, if this platform supports unmapping mmapped files.

    - #### UNMAP_NOT_SUPPORTED_REASON

          public static final String UNMAP_NOT_SUPPORTED_REASON

      if `UNMAP_SUPPORTED` is `false`, this contains the reason why unmapping is not supported.

  - ### Constructor Detail

    - #### MMapDirectory

          public MMapDirectory​(Path path,
                               LockFactory lockFactory)
                        throws IOException

      Create a new MMapDirectory for the named location. The directory is created at the named location if it does not yet exist.
      Parameters:
        `path` - the path of the directory
        `lockFactory` - the lock factory to use
      Throws:
        `IOException` - if there is a low-level I/O error

    - #### MMapDirectory

          public MMapDirectory​(Path path)
                        throws IOException

      Create a new MMapDirectory for the named location and `FSLockFactory.getDefault()`. The directory is created at the named location if it does not yet exist.
      Parameters:
        `path` - the path of the directory
      Throws:
        `IOException` - if there is a low-level I/O error

    - #### MMapDirectory

          @Deprecated
          public MMapDirectory​(Path path,
                               int maxChunkSize)
                        throws IOException

      Deprecated.
      use `MMapDirectory(Path, long)` instead.
      Create a new MMapDirectory for the named location and `FSLockFactory.getDefault()`. The directory is created at the named location if it does not yet exist.
      Throws:
        `IOException`

    - #### MMapDirectory

          public MMapDirectory​(Path path,
                               long maxChunkSize)
                        throws IOException

      Create a new MMapDirectory for the named location and `FSLockFactory.getDefault()`. The directory is created at the named location if it does not yet exist.
      Parameters:
        `path` - the path of the directory
        `maxChunkSize` - maximum chunk size (for default see `DEFAULT_MAX_CHUNK_SIZE`) used for memory mapping.
      Throws:
        `IOException` - if there is a low-level I/O error

    - #### MMapDirectory

          @Deprecated
          public MMapDirectory​(Path path,
                               LockFactory lockFactory,
                               int maxChunkSize)
                        throws IOException

      Deprecated.
      use `MMapDirectory(Path, LockFactory, long)` instead.
      Create a new MMapDirectory for the named location and `FSLockFactory.getDefault()`. The directory is created at the named location if it does not yet exist.
      Throws:
        `IOException`

    - #### MMapDirectory

          public MMapDirectory​(Path path,
                               LockFactory lockFactory,
                               long maxChunkSize)
                        throws IOException

      Create a new MMapDirectory for the named location, specifying the maximum chunk size used for memory mapping. The directory is created at the named location if it does not yet exist.

      Especially on 32 bit platform, the address space can be very fragmented, so large index files cannot be mapped. Using a lower chunk size makes the directory implementation a little bit slower (as the correct chunk may be resolved on lots of seeks) but the chance is higher that mmap does not fail. On 64 bit Java platforms, this parameter should always be large (like 1 GiBytes, or even larger with recent Java versions), as the address space is big enough. If it is larger, fragmentation of address space increases, but number of file handles and mappings is lower for huge installations with many open indexes.

      Please note: The chunk size is always rounded down to a power of 2.

      Parameters:
        `path` - the path of the directory
        `lockFactory` - the lock factory to use, or null for the default (`NativeFSLockFactory`);
        `maxChunkSize` - maximum chunk size (for default see `DEFAULT_MAX_CHUNK_SIZE`) used for memory mapping.
      Throws:
        `IOException` - if there is a low-level I/O error

  - ### Method Detail

    - #### setUseUnmap

          @Deprecated(forRemoval=true)
          public void setUseUnmap​(boolean useUnmapHack)

      Deprecated, for removal: This API element is subject to removal in a future version.
      Please use new system property `ENABLE_UNMAP_HACK_SYSPROP` instead
      This method is retired, see deprecation notice!
      Throws:
        `UnsupportedOperationException` - as setting cannot be changed

    - #### getUseUnmap

          @Deprecated
          public boolean getUseUnmap()

      Deprecated.
      use `UNMAP_SUPPORTED`
      Returns `true`, if the unmap workaround is enabled.
      See Also:
        `setUseUnmap(boolean)`

    - #### setPreload

          public void setPreload​(BiPredicate<String,​IOContext> preload)

      Configure which files to preload in physical memory upon opening. The default implementation does not preload anything. The behavior is best effort and operating system-dependent.
      Parameters:
        `preload` - a `BiPredicate` whose first argument is the file name, and second argument is the `IOContext` used to open the file
      See Also:
        `ALL_FILES`, `NO_FILES`

    - #### setPreload

          @Deprecated
          public void setPreload​(boolean preload)

      Deprecated.
      Use `setPreload(BiPredicate)` instead which provides more granular control.
      Configure whether to preload files on this `MMapDirectory` into physical memory upon opening. The behavior is best effort and operating system-dependent.

    - #### getPreload

          @Deprecated
          public boolean getPreload()

      Deprecated.
      This information is no longer reliable now that preloading is more granularly configured via a predicate.
      Return whether files are loaded into physical memory upon opening.
      See Also:
        `setPreload(BiPredicate)`

    - #### getMaxChunkSize

          public final long getMaxChunkSize()

      Returns the current mmap chunk size.
      See Also:
        `MMapDirectory(Path, LockFactory, long)`

    - #### openInput

          public IndexInput openInput​(String name,
                                      IOContext context)
                               throws IOException

      Creates an IndexInput for the file with the given name.
      Specified by:
        `openInput` in class `Directory`
      Parameters:
        `name` - the name of an existing file.
      Throws:
        `IOException` - in case of I/O error