Why do you need Apache Kafka for your cloud migration

1. Problem

Your company has decided to move to the cloud (AWS, Microsoft Azure, Google Cloud Platform, ...). Now what? Well, your cloud migration won’t happen all at once and it is usually a lengthy journey! You would need a strategy for hybrid integration between your on-premises and cloud deployments. In fact, there will be part of your applications and data sources in the cloud while others remain on-premises for some time. Some of your new applications will run independently in the cloud, while your other applications in the cloud will need data from your on-premises data center and vice-versa.

What strategy, architecture and best practices would accelerate such cloud migration, reduce costs and minimize risks and disruption to applications and data both on-premises and in the cloud?

2. Solution

I did advocate such strategy for hybrid integration between on-premise and cloud deployments in my own talk at the Hadoop Summit back on June 28th 2016. Please see the 'References' section below. It was more about sharing a few patterns of such hybrid integration and a call to companies to advance their own strategy for hybrid integration.

Later, I came across a solution proposed in a couple web resources and published by Confluent that shows how such hybrid integration would be implemented. This diagram from Confluent illustrates such solution. Please see the 'References' section below.

Since then, there are many advances in the Apache Kafka ecosystem that would make such proposed solution more implementable now more than ever:

• Kafka is available as a fully managed service on many cloud platforms. This makes the solution applicable not only to AWS, as initially suggested by Confluent, but also to Azure and to Google Cloud Platform as well.
• There is an updated version (2.0) of Mirror Maker that is used for Kafka replication. This version is supposed to address many of the limitations of the original Mirror Maker.
• There are many options available now for Change Data Capture especially as most data sources on-premises are still SQL databases.
• Kafka Connect, designed specifically for Apache Kafka as a framework for streaming data between Kafka and other data systems, now provides source connectors and sink connectors to almost every data system you would imagine including cloud-native ones!
• Kafka core and the other components of Kafka, such as Kafka Connect and KSQL, can be deployed on Kubernetes now. Such deployment offers many advantages out-of-the-box such as auto scaling, monitoring, better resource utilization, portability between on-premises and cloud, standardization on the same infrastructure, ...
• Both Kafka on premises and Kafka on the cloud have end-to-end monitoring capabilities to track event streams produced in both on-premises and cloud, see who is consuming them, find out what is the latency, ensure events were successfully replicated, … Such monitoring capabilities are offered by the Kafka service providers as well as third-party services.

The core idea of the solution is to build a central pipeline architecture that leverages:

• Kafka as Data Integration Hub between your on-premise data center and the cloud. You would need Kafka on premises and also Kafka on the cloud. You can host your own Kafka in the cloud or use one of the fully managed Kafka cloud services.
• Kafka replication between your on-premises data center and the cloud. Options for Kafka replication are Mirror Maker 2.0, uReplicator and Confluent Replicator.
• Fast private connection between your on-premises data center and the cloud. For example: AWS Direct Connect, Azure Express Route and Google Cloud Interconnect.

The combination of Kafka on premises, Kafka on the cloud and Kafka replication between the two, with a fast private connection, will provide you with a bridge between your on-premise data center and the cloud that enables fast and bi-directional data movement between on-premise and the cloud and also between different cloud providers.

3. Benefits 

1. Proven central-pipeline architectural pattern leveraging Kafka: Many companies successfully implemented such solution for their migration to the cloud. This solution would leverage proven tools to accelerate cloud migration, minimize risks and reduce costs.
2. Enterprise-wide strategy for cloud migration: Instead of each team and each department having their own cloud migration approach, the proposed solution provides an enterprise-wide strategy.
3. Recognition of the reality that a cloud migration won't happen all at once. In fact, applications and data stores will be on-premises and in the cloud ( or in multiple clouds) before a full migration is completed. 
4. Kafka will be a single event pipeline from on premises to the cloud and vice versa for all teams and applications. Otherwise, each team and each application might be building point to point pipelines that might be moving the exact same data. Each pipeline would require additional cost for development and testing and also operating cost for deployment, monitoring and maintenance. 
5. Bi-directional replication of event streams through Kafka from on premises to the cloud and from the cloud to on premises. This will enable applications in the cloud to get data from applications on premises and also enable applications still on premises to get data from applications running in the cloud. 
6. Support for incremental cloud migration where one application at a time is moved from on-premises to cloud without disrupting other applications still on-premises. For example, if such moved application continues publishing event streams to Kafka with the same schema, the other applications won’t even notice that it was moved to the cloud!
7. Continuous low latency synchronization between on-premise Kafka and Kafka in the cloud especially if a fast network is used with a properly configured Kafka replication tool. 
8. Multiple existing consumers: Both applications and data stores, can access the same event streams once they are replicated from Kafka on premises to Kafka on the cloud. This is achieved by leveraging existing Kafka Connect connectors to push data to different data stores without coding but simple configurations and built-in features such as scalability, failover, ... 
9. Kafka will be a single source of truth solving eventual data quality risks.  For example, separate pipelines might not provide the same level of reliability. There is a risk of one of these pipelines losing data and applications consuming such data won't have consistent view of the data.  
10. Centralized manageability and monitoring: A central event pipeline between on premises and the cloud simplify such operations especially if using a fully managed Kafka and Kafka on-premises with end-to-end monitoring capabilities to track event streams produced in both on-premises and cloud, see who is consuming them, find out what is the latency, ensure events were successfully replicated, …
11. Centralized security: A central event pipeline makes security easier to implement. For example, encrypting the event streams while they are replicated between on-premise and cloud would be a simple configuration of Confluent Replicator for example compared to applying encryption to multiple pipelines for multiple data stores and applications. 
12. Centralized governance: A central event pipeline makes governance easier to implement. For example, it will be easy to track where event streams are coming from and who is consuming them. 
13. Lower development and operational costs: Implementing a single pipeline and streaming every source of data to the cloud just once saves cost compared to multiple pipelines and multiple copies of the same data by cutting down cross data center traffic. Taking advantage of Kafka's built-in compression will further cut down cost. For new use cases, new producers can be quickly added to easily write to Kafka and new consumers can be added later to easily read from Kafka. 
14. Leverage the entire Kafka platform and Kafka Cloud Service to build complete streaming data applications. In addition to leveraging Kafka ( Kafka Core and Kafka Replicator) as a one-time data migration hub or a persistent bridge between an on-premise data center and the cloud, other components of the Kafka platform and a fully-managed Kafka cloud service can be leveraged as well. Such components are Kafka Connect pre-built connectors, Schema Registry, Kafka Streams and KSQL. They will help not only with the cloud migration but also leveraging Kafka as the basis for micro-services, real-time data flows, sophisticated stream processing applications and machine learning applications. All of this while benefiting from scalability, latency, uptime SLAs…
15. Architecture simplification: In a later phase, applications can be modified to work directly with Kafka and report the event streams to Kafka instead of data stores such as SQL databases. This will simplify the architecture ( no need of a CDC tool for example) and provides lower latency and makes Kafka a central bus connecting all applications. 
16. Dramatic simplification of the data flows between on-premises and cloud environments: Builders of an application can simply publish their event streams locally on premises or on the cloud. They don’t have to reason about additional things such other data flows, other geographical regions, consumers of their event streams and the failure modes where the cloud environment is not available from the on-premises environment or vice versa. 
17. Continuous operation in a hybrid mode (on-premises and cloud): You might still want to keep a specific application on premise due to multiple reasons such as cost for example and decide not to move it to the cloud. You might decide to migrate back an application from cloud to on-premises!
18. Multi-cloud integration: You might have a multi-cloud strategy where you need to use the services of two or more cloud providers. This could be driven by requirements such as Geo-locality, regulations, integration of an acquired company that opted for a different cloud provider, use of a specific service available or best at a particular cloud provider ...
19. Support of inflight transformations: Leverage Kafka Connect SMT (Single Message Transforms) to transform single messages as they flow from on-premises to cloud ( or vice versa) without writing code for masking, filtering, routing...
20. Support for polyglot persistence: Many applications use a combination of several different storage technologies for different purposes such as full-text search, data warehousing, application-level caches, ... Such approach is known as polyglot persistence.

Please share any insights if you implemented such central pipeline architecture in one of your projects. Your comments and feedback are much appreciated.

Thank you!

4. References:

• Analysis of major trends in Big Data Analytics, Slim Baltagi, June 28th 2016 Slides, Video
• Blog: Want to migrate to AWS Cloud? Use Apache Kafka, Gwen Shapira, Confluent September 22, 2016
• Bridge to the cloud: Using Apache Kafka to Migrate to AWS, Jay Kreps, Confluent, AWS re:Invent November 2017 Slides, Video