Shorticle 977 – Asynchronous multi-cloud storage replication using Google event-driven transfer
Data replication is common usecase to replicate database across multiple locations like cloud and on-premises or storage file replication across location for batch processing. Usually, data synchronisation for databases are achieved through platform built tools to handle real-time or batch synchronization.
For storage replication, we need to rely on pull mechanism to run scheduled batch script or functions?which periodically checks change in storage space and replicates to target storage when there is change in data in source storage location. This cloud be costly and time consuming as we need to run scheduled application like batch to monitor and replicate at regular intervals.
In cloud platform, this would not be effective as real-time data synchronization may get affected due to network latency and many other factors. Google cloud provides a Storage transfer service (STS) which can be used for real-time or near real-time data synchronization between storage services. This is effective for multi-cloud platforms to replicate from AWS S3 bucket to Google cloud storage and this could be useful for multi-cloud DR facility between AWS and Google cloud, multi-regional storage replication within Google cloud for high availability.
This STS facility is improved further with the recent announcement to introduce event-driven transfer for real-time storage transfer. It is a serverless facility to tightly integrate with Pub/Sub message bus for event trigger messages and notification service for event notification. This will be a push transfer as against the regular pull transfer earlier and it can be used for one or more consumers registered for storage replication.
During event trigger with STS, event stream runs job for storage replication across regions or location in Google cloud or storage service like AWS S3 in multi-cloud transfer. Event trigger requires pubsub subscription for event trigger, notification service for event notification and transfer job for event stream facilities.