Kafka Replication Protocol

In this article, I'm going to explain you the Kafka Data Replication Protocol with a very simple example.

[?]?Learning Outcomes From this article:

??? Kafka Data Replication Protocol

?? Concept of Watermarks to serve consumer clients with consistent data.

?? Leader Election during failure and Log Reconciliation.

Read the story ahead.

?? There's a classroom called 'Kafka', which has 3 students.

??? Johny Sins

??? KRK

??? Waqar Zaka (from Pakistani Rowdies). Smart dude.

These students need to make notes while attending lectures in classroom and 1 of them (the leader) will use their notes to answer the questions asked in examination hall.

But they have 4 core-responsibilities to abide by in Kafka School:

1. To maintain their OWN registers where they will make notes.

This is how the register would look.

1. Format for note-taking: (Leader-Student's name, Chapter-Number, Actual-Note).

Example: Say Johnny Sins makes notes on: How to do plumbing (which is chapter-1). Sample record:

3. In examination question paper, ONLY LEADER is allowed to answer a question and the condition under which they would answer the question:

IFF ALL STUDENTS in school HAD MADE NOTES FROM THAT CHAPTER IN THEIR RESPECTIVE REGISTERS.

4. All students are part of a set - 'Good students'. But if they misbehave, they will be deleted from this set. I will define misbehaving in next sections.

?? So let's learn three concepts of Kafka with above story characters:

[?] How followers replicate data from leader?

Leader: When you as a developer, produce data into Kafka - it travels to 1 of the brokers, which contains a replica of your topic's partition. This exact broker is called a Leader.

Followers: These are the brokers which have replicated copies of data in a topic's partition.

?? Let's say, Johnny Sins is chosen as the leader amongst the 3 students initially. Then, while the teacher is teaching, Johnny makes all notes in the above described format in his own registers.

?? Now KRK and Waqar (who are the followers here) have to copy the chapter-notes from Johny, because all 3 are friends and want to succeed together. The way they do so:

• They issue a request saying, "Hey, we have already copied notes from "Johnny" until "Chapter-3", so if there are further chapter notes, please send us those".
• Johnny looks at his own register and responds by providing the further chapter notes.
• KRK and Waqar conveniently copy those chapter notes in their own notebook.

That's how Kafka replicates data from it's leader to it's followers.

?? Technical Mapping:

All followers issue a fetch record request by sending a payload to the leader. The payload contains the last offset that the follower has successfully comitted to its log. The leader responds with the data beyond the last committed offset to the follower, so that data gets replicated.

[?] How Kafka determines what records can be read by a consumer?

Question: Let's say that in the exam question paper, there comes a question from Chapter-5. So, how would the leader (Johnny Sins, in this case) answer such a question?

Answer: Simple.

?? Johnny would just see, the last chapter till which all his batch-mates have made notes.

?? Example: (SEE FIGURE BELOW)When KRK and Waqar requested Johnny to provide him data from Chapter 5 during data replication.

?? Then Johnny would be assured that both these people have made notes till Chapter 4 and hence if there's ANY question about Chapter 1, 2, 3, 4 then Johnny would answer it, else it would reject the question.

?? Technical Mapping:

?? Basically a record / offset is said to be committed when all followers in In-sync-replica set (characterised by 'Good Students' set in this story), request for data BEYOND a particular offset.

?? This value is maintained as a high watermark value. Basically an offset value signifying that all client requests that request for data before this offset would be served successfully.

[?] How a new Leader is elected and How log reconciliation happens?

Scenario:

?? Let's say Johnny becomes ill suddenly and someone else needs to become the leader and needs to start making notes.

?? At this stage, we will remove Johnny from the set of 'Good students'. So now, 'Good Students' set contains, 'KRK' and 'Waqar'.

?? Technically, when a broker dies away, then we remove it from an In-Sync-Replica set for that particular topic-partition.

Questions:

In this situation 2 key questions arise:

1. How would a new leader be chosen amongst KRK and Waqar?
2. How would the notes in KRK and Waqars' registers change that were made when Johnny was still the leader?

Answer:

?? Answer-1:

?? Answer to first question is very simple. Since both KRK and Waqar are part of the set - 'Good students', any one of them can be chosen as the new leader.

?? This is because both of them have been making notes from Johnny's registers consistently in their own registers.

?? So, if there's a question that comes from a chapter that both KRK and Waqar are aware of, then any one of them can answer that question. So it makes sense that any one of them can be promoted to become the leader.

?? Answer-2: Most interesting part !!!!!!!!

?? Suppose, when Johnny was still a leader - KRK had copied notes from Johnny into his own register till Chapter 5 and Waqar till Chapter 3.

?? And suppose that the high watermark (the offset till which all followers are caught up, is 3).

I have written value as 'J-v1'. Recall, that while appending values into register students need to tag the leader-student's-name along with value. It will be useful NOW !!

?? Now, as Johnny becomes ill and can no longer make notes as the leader, we can promote anyone from the set of 'Good-Students' as the new leader.

?? Suppose we select Waqar as the new leader.

At this stage, all the data that would be appended to Waqar's notes would be tagged as:

This change because, recall the structure of appending the records. It had a field called, current-leader-student's-name. This field in Kafka is called Epoch Number and is used in Log Reconciliation.

[?] Log Reconciliation:

?? Now, when new chapter's notes are written by Waqar in his notes, then KRK would try to replicate those notes into his own notes.

?? Hence KRK would make a request to Waqar, saying,

Hey, I've made notes from Johnny's registers into my own register till Chapter 5. And I want you to provide me notes beyond this chapter please.

?? Hearing this, Waqar becomes furious !!!

Why?

?? Because, he's the new leader and is himself at Chapter-5 only and his follower (KRK) is at Chapter 8 (with some stale uncommitted data from an old leader (Johnny)).

?? So, at this stage - Waqar commands KRK to DELETE the stale UNCOMMITTED data (which was copied from the former leader (Johnny)).

?? At this point, KRK deletes the uncommitted data and asks for data beyond the last successfully committed chapter/offset.

?? So now, Waqar provides the new data to KRK and he is able to update his register with latest data from current leader.

Technical Mapping:

?? This deletion of data from a broker which is stale (essentially having uncommitted records from an older leader and then copying the latest data from the newly chosen leader) is called Log Reconciliation.

?? This is the fundamental base of many Data Replication Protocol. Not just in Kafka.

[?]?Conclusion:

?? Thank-you very much for making it thus far. Hope you learnt a thing or two about Apache Kafka, if you're a beginner :)

???Please comment if you have any questions or found something incorrect !!

???? Godspeed !!