What are the main challenges and risks of data modeling and data engineering in a cloud environment?

For most firms, cloud is an extension of their on-premise infrastructure, data and applications. Many firms operating in regulated industries have, over the past number of years, invested in teams and tools for data governance that are optimized for on-premise scenario, where-in data in Transaction Processing and Analytics & Reporting Databases, Data Warehouses and Data Marts was what was governed. In cloud, many firms start with data in file/blob stores such as S3 with a Data Virtualization layer on it. With transition to cloud, firms have to figure how to optimally extend or complement their current data governance investments.

While there are built-in tools for data governance and security in the cloud provider ecosystem, the configurations will depend on the company policies. This is more challenging when dealing with hybrid implementations.

Pre-cloud, data modeling and data engineering would primarily focus on Databases, Warehouses, Marts. With some difficulty, they might extend their reach to cover ETL, Reports and Dashboards. The coverage would likely be just for batch scenarios. Data moving through message buses or point-to-point integration (APIs) would rarely be the remit of data modeling and interoperability functions. With migration to cloud, the business expectation often is that batch integration would quickly become continuous. Use of services such as kafka instead of or in addition to batch oriented ETL pipelines quickly becomes the norm. This can be a significant change in scope from a data modeling perspective.

Every major cloud provider offers many services with significant differences in how they charge for using them. Security considerations can potentially impact performance and to compensate, services can be scaled up, but costs will scale up alongside. Hence optimizing performance, consumption and costs associated data on cloud is very important but non-trivial.

Data is a critical asset. In an increasingly digital operating environment, monetizing this asset is a key competitive necessity. In reality, different businesses/ functions understand and own parts of this asset. While there are CDO/CDAO functions, data in most medium and large enterprises is quite complex and it would not be incorrect to say that no one understands the whole and can effectively govern the whole. Hence, data ownership & governance is often federated to respective businesses/ functions. This creates two challenges, those looking to innovate and experiment data from across these silos need to get access from multiple owners and understand the nuances/ significant differences in data between areas that may have the same name.