Event Driven point of view for Distributed Systems

Lets take an example of state of the art Hadoop Distributed File System.

1. There is name node and data node.

2. When you save a file on HDFS, that file gets splitted into blocks, each block 128MB according to Hadoop 2.0

3. Each block gets replicated 3 times, considering block default replication factor of 3.

4. Consider my file has total 5 blocks. Those 5 blocks becomes 15, each block replicated 3 times.

5. Those 15 blocks gets distributed to data nodes. Name node keeps the metadata of that file, on which node the blocks and replicas are stored. Blocks replicas created for fault tolerance.

Now, how does name node knows if data blocks are alive?

Every data node sends the heart beats every 3 seconds name node, to let him know that "I am alive". Ha ha ..Sounds interesting, right?

If name node does not get 10 consecutive heart beats, then it assumes that the respective data node is dead.

Replication factor 3 for some of the files stored on dead data node goes down to 2. Name node then executes the fault tolerance mechanism & scales it to 3 again by creating the replica of respective blocks on alive data node. Name node then updates metadata.

To conclude, this is how any distributed systems can be designed using Event Driven Architecture. Events generated in case of HDFS are heart beats to let know that "I'm alive"

Follow me or share connection request if you would like to learn more from me.