What are the key principles for designing data architecture in serverless applications?

The Data architecture must be informed by enterprise architecture , strategy and vision. Data architecture should cover the system/capabilities landscape that process the data within a defined . When we talk about serverless, the infrastructure part is less important as the horizontal and vertical scaling is managed within the serverless component

Serverless applications are designed to run without managing the underlying infrastructure. Data architecture in serverless applications should follow these key principles: 1. Use simple and fast functions that perform single-purpose transactions. 2. Optimize for concurrency and scalability by keeping servers stateless. 3. Use temporary storage for local resources and persistent storage for durable data. 4. Write hardware-agnostic code and dependencies. 5. Orchestrate workflows with state machines and events, not functions. 6. Design for failures and duplicates by using idempotent operations and retries.

Outlining data requirements stands as a pivotal stage in shaping data architecture or models. It involves recognizing factors like file format, data volume, frequency, and type, essential for constructing resilient and versatile data pipelines in the long term. Identifying these aspects contributes to the development of a comprehensive and durable data architecture that supports generic and robust data processing pipelines.

Before defining the data architecture, the data requirements like pipelines, sources, frequency, formats, volume, variety, business needs, security rules, and regulatory constraints need to be carefully assessed. This assessment will help in defining and choosing the right data architecture considering the ingestion, storage, processing, scalability, integration, visualization, and analytics needs of stakeholders.

When designing data architecture for serverless applications, start by thoroughly understanding your data requirements, including sources, types, volumes, frequencies, and formats. Align these requirements with your business objectives, user needs, and any regulatory constraints to inform your choice of data storage, processing, and integration solutions. This foundational step ensures that the architecture is tailored to effectively support the specific demands and goals of your serverless application.

The new "idea" in data space is to have decoupled storage and compute. What this mean is that the data storage is not tied up to a processing system like a database or data warehouse. The data is "portable" and can be moved across storage platforms and does not lie within the procesesing sytem. Delta lake and Iceberg are two examples of decoupled storage and can be used in serverless architecture along with mentioned options like Amazon S3, DynamoDB, Aurora Serverless, and Firebase.

A good approach would be to use multiple different types of Data storage options. There's no One magic schema sauce for all. - Want system with high write throughput ? RDBMS is your go to option. - Wanna run read-intensive queries typical in data warehousing columnar database is a good option. A tradeoff with these type of databases are they might not be the optimal choice for OLTP (Online Transaction Processing) scenarios with frequent updates or inserts. - Document databases are suitable for scenarios where data is semi-structured or unstructured. They provide flexibility for storing diverse data types within documents.

Object data stores will be always far more economical compare to relational data bases. And it is also possible to define table structures on top to objects using Apache Iceberg or similar specifications. Databrick’s delta table format is another known option.

The data is stored ideally in 3 stages according to my perspective: 1. Raw : Where data is directly dumped from Source without any transformation. 2. Staging : This layer cleans raw data and does data massaging to remove certain anomalies. 3. Consumption : This layer contains transformated data that is ready to be consumed by the Data Visualization team for drawing out charts/ Visualisations.All derived columns / computed values will be calculated in this phase. For all 3 layers it is ideal to get a object storage like s3 as it will be cost effective also readily available for any systems for consumption via aws services like AWS DMS, AWS glue. You can also store the layers into different schema (Incase of RDS) of the target database.

In selecting data storage options for serverless applications, weigh the pros and cons of different storage types like relational, non-relational, object, and file-based systems. Evaluate key factors such as availability, durability, scalability, and performance to match your application's needs. Consider popular choices like Amazon S3 for object storage, DynamoDB for NoSQL, Aurora Serverless for relational databases, and Firebase for real-time data and file storage.

The most important aspect is the introduction of modular and event-driven structures that ensure efficient data ingestion, transformation and delivery. The most important aspect is the development of pipelines that scale efficiently with changing workloads and match the dynamic nature of serverless applications for seamless data processing. From personal experience, I would recommend using Databricks for all data engineering tasks, such as data pipeline design. Databricks offers collaborative notebooks, interactive queries and MLflow integration (interesting for generative AI pipelines) and improves the end-to-end capabilities of the data pipeline through a unified and collaborative analytics platform.

Some points following in design data pipelines: ?? Define parameters of concurrency and retries for serveless processing. ?? Define processes that control failures and try to solve problem itself or alert it and reflect it in monitoring tool. ?? Choose serverless tool to specific operation, for example: ??Lightweight processing ? AWS lambda and kinesis firehouse ??Integration process ? AWS glue. ??Queue ? AWS SQS and kinesis data streams ??Storage ? AWS S3

You can design both synchronous and asynchronous data pipelines which could either process batch data (file-based data at regular schedule or one-time adhoc) or streaming data (real-time or near real-time in nature) in serverless application. Synchronous data pipeline:- API Gateway -> AWS Lambda -> S3 Pre-signed URLs for access. Asynchronous data pipeline:- S3 event () -> SNS (publisher) -> SQS queues (various queues as subscribers) A well-designed system must have error-handling (SQS dead-letter queues), notification (SES error emails or SNS texts), governance (datalake and IAM Policies and S3 bucket policies), and security (SSO-based or AWS Cognito User pool based authentication and authorization).

From my Spark tuning experiences ??: ?? Avoid UDFs: They're not always the heroes. Find friendlier alternatives for a smoother ride. ?? Shuffle Smartly: Spread your data wisely among executors to keep things running smoothly. ?? Cache with Care: Stash away costly computations, but don't forget to tidy up – storage has limits. ?? Multithreading Magic: Make use of multiple threads for a turbocharge within executors. ?? Keep Data Neat: Well-organized data means fewer headaches, especially with the tricky skewed stuff. ?? Broadcast: Spark's broadcasting is like a superhero team-up for handling shared data effortlessly and boosting performance.

Monitor and analyze metrics and logs to detect bottlenecks, errors and anomalies. Use caching, compression, partitioning (e.g. Databricks' new Liquid Clustering), indexing to improve data access and processing speed. The critical aspect is the continuous refinement and optimization of data performance to ensure that the serverless application operates efficiently as workloads and requirements change. Use Databricks Delta for high-performance analytics and data processing. The Delta engine accelerates query performance and provides advanced indexing and caching capabilities. First and foremost, ensure that the data architecture is tuned for scalability and reliability to enable consistently high performance in a serverless environment.

As per my experience, optimizing Big Data is very crucial for enhancing the data transformation speed. Techniques such as Data Caching, Partitioning/bucketing, and utilizing efficient data formats like Parquet and ORC significantly accelerate processing times. As per my recent discovery for data optimization techniques one of the best method was AQE (Adaptive Query Optimization) in Spark 3.0. It dynamically optimizes code during runtime, resulting in substantial improvements, saving both time and costs in data processing.

Optimized data usage can improve query performance, reduce resource consumption, and enhance overall system efficiency. Profile data. Understand your data and ensure that your data model is well-suited for your workload. Fine-tune your data storage configuration. Configure your data storage infrastructure to align with your workload requirements Optimize query performance. Analyze and optimize queries that run in the workload. Regularly monitor and tune the system. Continuously monitor the performance of your workload and iterate on the data storage configuration and query optimizations.

You should use diagrams, schemas, metadata, and code comments to describe the data flows, structures, relationships, and dependencies in your serverless application. You should also use version control and documentation tools, such as Git, GitHub, or Google Cloud Source Repositories, to track and manage the changes in your data architecture.

From my experience and practices, Confluence is a good way to go for documenting data architecture stuff. It's like a virtual whiteboard where the team throws in diagrams and code bits. Plus, the version control keeps things from going haywire. It's been a game-changer in making data architecture docs less of a headache. While code repositories are ideal for collaborative coding, a smart move is to embed Confluence document URLs in readme files or code comments. This practice seamlessly connects documentation with the codebase, keeping things well-linked and accessible.

Make adjustments based on feedback, changing requirements and new opportunities. The key aspect is to continually refine and adapt the data architecture to ensure it is responsive to dynamic application changes and consistently aligned with evolving business objectives. I myself am used to relying on Databricks Auto-Optimization for automatic performance improvements: Databricks Auto-Optimization uses machine learning to fine-tune workloads and seamlessly optimize data processing and analytics performance in a serverless environment. Maintaining agility is also an important aspect to quickly adapt the data architecture to changing application landscapes and ensure continuous alignment with business goals and performance optimization.

You should use diagrams, schemas, metadata, and code comments to describe the data flows, structures, relationships, and dependencies in your serverless application. You should also use version control and documentation tools, such as Git, GitHub, or Google Cloud Source Repositories, to track and manage the changes in your data architecture.

Architecture of serverless applications should ideally be flexible to change the architecture as the product or data matures aiding lesser time for various DataOps

Evolvability in serverless data architecture means creating a flexible system that can adapt to future changes, like business needs or technology updates. This involves building modular, independent components for easy updates, designing for scalability, incorporating automation, and utilizing cloud-based managed services to handle maintenance, enhancing the system's adaptability.

The most important part to consider for Serverless Data Architecture is Event Driven. You need to execute certain tasks only after some previous task is completed successfully. We can use AWS EventBridge, AWS SQS, AWS Lambda to make our architecture Event Driven on AWS and make our pipelines more robust. We also use AWS SNS to get the alerts if certain steps fail and we can immediately resolve.

In my experience Data Architecture should also address how Data Quality/governance, Data Lake/Warehouse storage and ingestion, serving data for AI/ML and Data science needs and for operational needs. Else all these tend to become siloed topics with siloed solutions and there is no cohesive system that works effectively. Some of these decisions may fall under the domain of Data Engineering instead of Architecture.

? Remember that any approach defined about data architecture in industry can support serverless approach: it's separate development of application without worry about the infrastructure will execute it. Some points help to implement it are: ?? Decoupling storage from processing. ?? Auto-scaling servers. ?? Containerized of code.

Contributing to a successful Data Model/Architechture is adding Data Quality and Data Governance check for PII or other important datas. Data Reconcilaiation helps us to make sure we are not dropping or missing any valuable data which might be needed for further KPIs. Transforming data is easy but making sure it withstand all type of data is difficult. So UTC(unit test cases) is important to analyze and make sure our data pipline is generating the data as expected

Security: Security in serverless data architecture involves incorporating strong measures from the design stage. This includes data encryption, careful access control management, regular security audits, secure coding practices, and automated security checks. The aim is to create a resilient system that effectively protects sensitive data.