Modern Data Stacks

Since my passion and work revolves around data and analytics, I decided to check out the latest conference on Modern Data Stack by Fivetran.

I have summarized my understanding and perspective after checking out the keynote event “An Executive Eye on the Future State of Analytics” at the conference

?1. Key changes in last few years:?Everyone connected to the data & analytics world will know this, “Infrastructure Revolution” or “Move to Cloud” which brought in cost reduction, reduction in management overhead and faster time to market.

2. Expectation in next few years:?

• Friction between personas and technologies to reduce: currently there is a rigid set of expectation on how different personas (Data Scientist, Data Analyst, Data Engineers and more) will interact with data and with each other, which would become much more fluid in future.
• Responsible AI: how do you ensure data is used in the right way
• Data workload: to increase, due to mainstream (enterprise-wide) adoption of AI use-cases

3. Closed Vs Open Models:?Analytics world has been taken over by the separation of Compute and Storage layers. But there seems to be different camps on whether it needs to be a closed or open model (where customer has a better view and control over ways of storage and compute). In the long-term I believe this should go the open way and it was summarized neatly by Ali (Databricks) & Rohan (MSFT),

• Open models will help in the long run as lot of innovation happens in the storage/file format world on open source side and it is easier and faster to incorporate these changes in Open models
• Paying SQL rates for other types of workloads (like AI) might not make sense, where as they should/could be charged at cloud storage retrieval rates (like S3, ADLS rates).

P.S:?This is my understanding and perspective about the keynote event which might not be accurate or what the presenters intended, so kindly check out the conference videos for a more comprehensive view.