Tuning and Optimizing memory for Elasticsearch

• Memory is one of the most critical resources to monitor in elastic search.
• The most common error that arises in elastic search is Out Of Memory.
• This error shows when the node cannot cope with the required heap size space.
• To avoid this you need to monitor the heap utilization and garbage collector performance closely.
• Most operating systems try to use as much memory as possible for file system caches and swap out unused application memory. This can result in the JVM heap or even its executable pages being swapped out to disk. It can cause garbage collections to last for minutes instead of milliseconds and can cause nodes to respond slowly or even to disconnect from the cluster. This can be resolved using one of the three options mentioned below, all three reduces the kernel’s tendency to swap:

Disabling all swap files

sudo swapoff -a         

Configure swappiness by setting the sysctl value vm.swappiness is set to 1

vm.swappiness=1        

Enable bootstrap.memory_lock, this locks the process address space into RAM, preventing any Elasticsearch heap memory from being swapped out

bootstrap.memory_lock: true         

You should not allocate more than 50 % of the total RAM to the JVM heap size. Elasticsearch requires memory for purposes other than the JVM heap. For example, Elasticsearch uses off-heap buffers for efficient network communication and relies on the operating system’s filesystem cache for efficient access to files. The JVM itself also requires some memory.

We tend to think that the more the memory the better the performance. HEAP memory is an exception. In any case, the heap size 26GB should be safe in most of the systems but not be set to more than 30GB. The JVM uses HEAP memory for storing object pointers - Compressed Ordinary Object Pointer (OOP). Above 32 GB of HEAP, Java needs to use regular, 64-bit, pointers. Because a lot of space will be used by pointer itself it drastically decreases how many objects can be stored in HEAP. Around 50 GB of HEAP will be the same as around 30 GB.