HBase MemStore

HBase MemStore

All the updates in memory as sorted KeyValues are stored in the MemStore. Basically, Data which contains sorted key/values is stored in an HFile. Moreover, per column family, there is one MemStore. Also, all the updates are sorted per column family.

What is HBase Memstore

In other words, all the in-memory modifications to the Store generally stores in a Memstore. Here, modifications are KeyValues.

Make sure we should not call the functions of HBase Memstore in parallel.

Let’s revise features of HBase

Some Key points or Memstore in HBase:

• In simple words, before a permanent write, a write buffer where HBase accumulates data in memory is what we call the MemStore.
• While the MemStore fills up, its contents flush to disk to form an HFile.
• It forms a new file on every flush, rather than writing to an existing HFile.
• Basically, for HBase, the HFile is the underlying storage format.
• Per column family, there is one MemStore. It is possible that one column family can have multiple HFiles, but not vice versa.

Following occurs, while the server hosting a MemStore that has not yet been flushed crashes:

• In order to record changes as they happen, every server in HBase cluster keeps a WAL. On defining a WAL, it is a file on the underlying file system. However, until the new WAL entry is successfully written, a write isn’t considered successful, this explains its durability.
• The data which was not yet flushed from the MemStore to the HFile can be recovered by replaying the WAL, if HBase goes down, that is taken care by Hbase framework.

Uses of HBase MemStore

Well, HBase users and/or administrators must know the meaning and uses of HBase MemStore, because:

Learn Hadoop from Industry Experts

Have a look at uses of HBase

• In order to gain better performance as well as to ignore issues, we can use MemStore in HBase. However, it is not possible to adjust settings in HBase on the basis of usage pattern.
• However, make sure constant flushes of HBase MemStore can affect reading performance in MemStore. Also, it can bring an additional load to the system.
• Moreover, the way in which MemStore flushes work it may affect our schema design.

Configuring MemStore Flushes

However, two types of groups are there of configuration properties in HBase MemStore:

• The first one determines at what time flush should trigger
• And, the second one also determines that at what time flush should be triggered but along with the updates which should be blocked while flushing.

Now, let’s learn about these groups in detail:

a. First Group

Basically, the “regular” flushes which happen in parallel with serving write requests, the first group triggers them.

However, for configuring flush thresholds, the properties are:

Let’s learn HBase Commands

• hbase.hregion.memstore.flush.size

1. <property>
2. <name>hbase.hregion.memstore.flush.size</name>
3. <value>134217728</value>
4. <description>
5. Memstore will be flushed to disk if size of the memstore
6. exceeds this number of bytes. Value is checked by a thread that runs
7. every hbase.server.thread.wakefrequency.
8. </description>
9. </property>

• base.regionserver.global.memstore.lowerLimit

1. <property>
2. <name>hbase.regionserver.global.memstore.lowerLimit</name>
3. <value>0.35</value>
4. <description>Maximum size of all memstores in a region server before
5. flushes are forced. Defaults to 35% of heap.
6. This value equal to hbase.regionserver.global.memstore.upperLimit causes
7. less possible flushing which occurs when due to
8. memstore limiting, updates are blocked.
9. </description>
10. </property>

b. Second Group

Well we can say, mainly for safety reasons, the second group of settings is there, like- there are times when write load is so high, that even flushing cannot keep up with it hence, for that writes are blocked unless MemStore has “manageable” size as we don’t want memStore to grow without a limit.

So, with following, it is possible to configure thresholds:

Explore Apache HBase Career Scope With Salary Trends 2018

• hbase.regionserver.global.memstore.upperLimit

1. <property>
2. <name>hbase.regionserver.global.memstore.upperLimit</name>
3. <value>0.4</value>
4. <description>Maximum size of all memstores in a region server before new
5. updates are blocked and flushes are forced. Defaults to 40% of heap.
6. Updates are blocked and flushes are forced until size of all memstores
7. in a region server hits hbase.regionserver.global.memstore.lowerLimit.
8. </description>
9. </property>

• hbase.hregion.memstore.block.multiplier

1. <property>
2. <name>hbase.hregion.memstore.block.multiplier</name>
3. <value>2</value>
4. <description>
5. Block updates if memstore has hbase.hregion.block.memstore
6. time hbase.hregion.flush.size bytes. Useful preventing
7. runaway memstore during spikes in update traffic. Without an
8. upper-bound, memstore fills such that when it flushes the
9. resultant flush files take a long time to compact or split, or
10. worse, we OOME.
11. </description>
12. </property>

Read Complete Article>>