The Stream than storms (Kafka)

Kafka has been a revelation over the years and has scaled up very well in establishing its capability as proven scalable distributed streaming platform. In today's modern world of the FAD of micro services application, it was highly essential to build applications which can integrate with each other reliability during real time transactions and also transform / react to streams of data. This technology provides a good answer for both these complex situations.

Kafka as a platform provides a scalable, clustered architecture which ensures the infrastructure to retain the data you publish to it and automatically balance the partitions in transparent manner to the applications which want to publish or consume data from the platform. It also provides the API to applications to extend the platform for the interactions and help them to easily integrate with each other as long as there is common understanding between them.

It manages the entire data you put in for it to hold using a simple key value pair combination which is the age old technique. But this also enables us to leverage this simple key value pairs to be used in following three important ways

• As Messaging Service - Scales well for multiple subscriber, highly reliable, available, supports consumer groups (scaled micro services) to read single streams
• As Storage Service - Being the distributed, immutable messaging system it has aided to be used as Storage system for anyone to query the data and manage this more like Tables if required. This is further supported by community Confluent which is actively packaging and supporting this platform and extend the same for better application.
• As Streaming Service - Given the vast amount of data it can hold, there is always need for applications to read them and derive some logical aggregations required. This provides capability to perform such complex aggregation via the API's available for extension which makes this more intelligent stream processor.

While this has been utilized largely by multiple real time applications and product companies eg. Twitter, JP Morgan etc where the use cases and volume of transactions are plenty, there are banks who have already come forward and leveraged them very effectively. Most of them are show cased as part of DevOx conferences available over the web, that I have read about multiple industry like Media (New York Times), Manufacturing (Audi) the following are the key ones that struck my attention towards transformation of financial industry.

RABO Bank - Europe

Confluent has worked with Rabo to enable a trial of moving the alerts going to customers from traditional mainframe platform level alerts to make them realtime using Kafka. The bank with the aim to see the entire architecture to be more scalable and flexible which is the focus for most of the financial industry in journey to cloud this is definitely a credible start given the security playing heavily in financial industry.

Royal Bank of Canada - Story of Canada

They have implemented Kafka as central nervous system. They have freed data from legacy mainframe systems over the last few years. Today they have 30+ use cases in 50+ applications across 10 different business on this streaming platform. It powers from transformation from monolith to cloud native, reduce analytic lead time from week to seconds, share common data records cross business units and also collaborate with external partners with single source of truth. They are now moving towards the vision of being event driven bank.

One Bank - ING - Europe

They have grown the trust over a period of time. They are moving payments platform to Kafka which is relatively a backbone of banking and this one reaffirms the ability of scale and reliability. ING being the digital bank with One Bank vision and this large enterprise has been leveraging this technology as common thread to tie this organization.

With what the way it has shown towards cloud enablement, analytics application, integration capabilities, api extensions I am sure this platform is going to storm the digital transformation journey across industries. The community is also heavily active and makes tremendous contributions in this area.