Kafka Gotchas
I've assisted several large clients in building a microservices-style architecture using Kafka as a messaging backbone, having a reasonably good understanding of its abilities and the use cases that really bring them out. But I'm not a Kafka apologist by any stretch; any technology that has gone through such a rapid adoption curve is bound to polarise its audience and rub certain developers up a wrong way, and Kafka is no exception. Like anything else, you need to invest a significant amount of time in getting across Kafka and event streaming in general, before you become fully proficient and can harness its might. And be prepared to face one or two frustrations, to put it mildly, along the way.
I've compiled a list of shortcomings that may cause developer frustration, or catch out unsuspecting first-timers. In no particular order:
Too many tunable knobs
The number of configuration parameters in Kafka can be overwhelming, not just for newcomers but also seasoned pros. Possibly with the sole exception of the JVM, I cannot think of another technology that has this many configuration parameters. This isn't to say that the config options aren't necessary; but one does wonder how many of those parameters could instead be replaced by ergonomics, much like Java did with G1. So rather than specifying a plethora of individual thresholds and tolerances, let the operator set a performance target, and have the system derive an optimal set of values that best meet this target.