Building a Data Mesh Platform: ChatGPT as a Game-Changer for Data Product Delivery

Introduction:

As the world increasingly depends on data to guide decisions, businesses must adapt to remain competitive. With the rapid evolution of data engineering and data science, leveraging the appropriate tools and strategies for efficient data consumption is crucial. We have embarked on a journey to construct a state-of-the-art data mesh platform. One of our objectives is to render data products more accessible and valuable to our users.

What if ChatGPT Served as an Output Port for Data Products?

As we deliver data products, we are exploring innovative methods to make data more accessible and valuable to our users. One potential solution could be ChatGPT, an AI language model created by OpenAI.

ChatGPT can process vast amounts of data, extract insights, and convey those insights in a natural language format. By integrating ChatGPT into our data mesh platform, we can offer users a robust, conversational interface for engaging with data products. This would empower them to:

• Ask questions and receive relevant, concise responses
• Generate reports and summaries on demand
• Identify trends and anomalies in data
• Access real-time analytics and make data-driven decisions

Enabling New Possibilities

As we advance in developing our data mesh platform, integrating solutions like ChatGPT presents numerous advantages:

1. Enhanced User Experience: ChatGPT enables users to interact with data in a more natural, conversational manner, lowering the learning curve and making data more approachable.
2. Increased Data Literacy: By providing a user-friendly interface, ChatGPT encourages users to engage with data products, promoting data literacy throughout the organization.
3. Improved Collaboration: ChatGPT can help bridge the gap between technical and non-technical team members, fostering better communication and collaboration around data-driven projects.

How Could This Work?

To assess the current state of GPT models, we have built a small demo based on the Titanic dataset, available on Kaggle. We employed chain-of-thought prompting to evaluate human queries on datasets. All prompts below used ChatGPT 3.5 directly out of the box and connected the Titanic dataset using the Pandas agent provided by Langchain.

For instance, we asked, "What is the average age of the people who survived on the Titanic?" ChatGPT was able to accurately translate this into Pandas code that returned the correct age: 28.34 years.

Simple retrieval questions like "Was there a Fernando on board?" resulted in "No" or "What can you tell me about the oldest passenger on board?" led to "The oldest passenger on board was a male, aged 80, named Barkworth, ...". We tried more advanced queries and all were answered successfully!

Granting ChatGPT access to an agent that it can use to query data results in an entirely new way of utilizing these models. It reduces the need to hallucinate and (in this small demo) was capable of basing its analysis on facts.

What Do I Think This All Means?

As we progress further in our journey toward a data mesh architecture, we have delivered numerous data products which are well-defined, documented, readable, and easily accessible for end-users. Coincidentally, this is the ideal input for ChatGPT to begin interacting with the data, as this makes it straightforward to include all relevant information, context, and understanding in prompts.

Further Steps

All concepts and use cases are still in very early stages, and a lot of unknowns remain. I am highly enthusiastic about the possibilities while also remaining mindful of the risks this technology poses. I would love to hear your thoughts in the comments, especially if you are also interested in this sort of use case!