You've migrated to a cloud-based data architecture. How do you tackle performance issues that crop up?

When moving to Cloud based data architecture following actors are at play - 1. Compute - How much compute is required to compute real-time, hourly, daily workloads ? 2. Storage - What storage is being used for each workload - Persistent and Ephemeral, Database technology 3. Network - How is data transferred within network for different workloads 4. Application Architecture - Data partitioning strategy, big data stack etc Optimization for performance will need careful evaluation of these levers with respect to Instance configurations, storage types and network technology selection for each workload. Not every workload needs to be performance optimized, as costs do matter so clear identification of high performance workloads is needed.

The best way to solve a performance problem is not allowing it to occur at the first place. Having said that , It is easier said than done so there are some things you should do proactively and also set up standard operating procedure to tackle those problems.First thing is to measure the performance of jobs and create a report/dashboard so it can be updated automatically. Once you see the trend of those jobs, Tackle the ones which are mostly long running or costing you more money. It is important to look at things holistically so you understand where to focus your energy and resources. It is also important to understand whether the juice is worth the squeeze . In other words, ROI of optimizing performance should justify the effort.

Usage of Monitoring Methods: Synthetic Monitoring: To proactively spot performance problems, mimic user behavior. Log analysis: To identify the underlying reasons, correlate log data with performance measures. Anomaly detection: Use machine learning techniques to spot odd trends. Do you want to learn more about a certain monitoring method or examine particular monitoring tools? Additionally, I would like to emphasize that you may greatly increase the resilience of your system by integrating automated cleanup with your monitoring efforts. Automating answers to frequently occurring performance problems can help you speed up problem solving and minimize manual involvement.

Treat your data platform as a factory: after all, it follows the idea of a factory turning raw materials or parts (input) to a finished consumable product (output). This transformation is where you add value. So, many of the ideas of successful factory management (Kaizen, product thinking, balance of processing) aimed towards optimising the flow of data from source to consumer apply here as well. This leads to resource optimisation as your processes are optimised: if not, apply your Kaizen to the transformations and your resource usage will drop to the right level.

Some important considerations are 1. Indexing optimally 2. Extensibility of data model 3. Tightly considered ACID criteria.

Yes, this is one of the key factors to fine-tune the new platform based on usage and need, which will lead to maximised performance of the data platform and minimize cost through enhancement of the data products and services and removing those which are not used.

Typical data tuning mechanisms like Indexing, Query Optimization, caching of data, Database Schema Design, normalization, denormalization, and query partitioning. Apart with in Database, we can also check and monitor with Hardware and Configuration of Database , in case hosted in an IaaS model we can have workload analysis and regular patch management of both Application and Infra. Regular performance testing and reporting helps us to maintain the data and database

The plethora of technologies out there is confusing, but this does offer a 'horses for courses' selection at the point of consumption. However, the backbone will be object storage as this is the cheapest and simplest way of managing data at a terabyte scale that many organisations now need. So, choose a data lake storage mechanism according to technology vendor of choice, then flex the remaining technologies for consumers according to need.

Do not underestimate the latency associated with this approach and keep the problem of eventual consistency in mind at all times. Instantaneous data everywhere is not possible, even with quantum computing, so localising data for local transactions is necessary.

Be sure to determine those factors that define success for you: in my experience it is a balance between execution time for a given process or end-to-end update time for a given source dataset versus the cost of execution when using cloud services. Often a fast execution will consume a lot of CPU, pushing your AWS or Azure (or GCP) bill up.

Leverage cloud provider-specific features like reserved instances, spot instances, and managed services to optimize costs and performance. Choose what fits best for your requirement.