Database Replication Pattern
Bhanja Kishore Biswal
Solution Architect at Infosys ,managing Devops,CI/CD, BigData,SnowFlake and AWS Cloud based applications, passionate in AI ,Deep learning
At defined intervals, CloudWatch (CW) triggers a Step Function (SF) that starts the Data capture using Database Migration Service (DMS)
?DMS connects to the database and captures data from archive logs stored in tables
DMS saves data as files in a raw bucket Asynchronously, CW invokes a SF which triggers the Glue Job in charge of data processing
The received files are processed so to reflect changes on existing database, optimize storage and access performance.
?Periodically, a Glue Crawler introspects the structure of the processed files to detect changes and update the Glue Data Catalog (GDC)
The GDC exposes the processed files as a database that can be queried via the Athena service
DevOps and Cloud Consultant
1 年Nice one.. Haven't tried this way