Shorticle 129 – AWS DataSync
Some common usecases in Fintech industries requires data synchronization between On-premises data store to Cloud data store like Reconciliation, reference data synchronization, user principle synchronization, workflow status updates, reporting schedule updates to name a few.
In such cases, batch processing using Glue job during End of the day and establishing data pipe between On-premises to Cloud would be a costly operation and hence to handle such a situation, AWS offers DataSync, which helps to get data updates through exported data feeds from On-premises to Cloud storage.
Image Source: AWS Documentation
In such a solution using DataSync, user has to develop a batch job to get delta updates from On-premises data store to be exported to data feed files and placed in Landing zones which gets pulled by DataSync agent and transferred securely through to Cloud using DataSync service and placed in AWS Storage services like S3 bucket or Elastic File System (EFS) or Windows File Service (FSx).
We can write a Lambda job to use this data feeds to update the data store in Cloud database as needed with flexible logic to overwrite or to validate and update the data. DataSync works as an Object store pipe to transfer data from On-premises to Cloud store. It is secure, self-managed and cost efficient for hybrid data synchronization.
For further read: https://aws.amazon.com/blogs/aws/aws-datasync-adds-support-for-on-premises-object-storage/
#magtechbytes #wipro #shorticle #shorticleaws