ChatGPT and the Struggle to Integrate AI into Applications

Like every other technical product-person in the world, when ChatGPT emerged, I was cranking through ideas for how it impacted our product and how it could be used.

ChatGPT was revelatory both inside and outside the AI industry. Outside the industry (where I self-identify), it revealed orders of magnitude improvement in capability over any previous systems we’d used out in the wild. Like most people, I’ve been driven crazy by Alexa and Siri and all the incredibly crappy rule or ML-driven Chatbots on websites. Unlike most people, I also have plenty of experience trying to implement natural language interfaces into BI tools and with advanced ML for a variety of applications from data quality to query performance to segmentation and behavioral prediction. But nothing I’d ever seen stacked up to the capabilities displayed by this new generation of Large Language Models. Internal to the AI community, I imagine that ChatGPT was almost equally revelatory; perhaps less for its capabilities than the way it electrified the world and generated a fanatic and widespread user-base.

I would imagine that every BI product professional in the world played around with ChatGPT and we all came to the same conclusion: traditional BI is dead. Why would anyone generate charts or table reports from data when they could just ask questions? And why, in god’s name, would anyone learn SQL to query data when A) ChatGPT can write SQL and B) ChatGPT could just answer the question if you didn’t need the raw data. Sure, people have been saying that for years. But until ChatGPT, the just ask questions part never worked.

So, like every other BI product team around the world, we at Digital Mortar started integrating ChatGPT into our product. That’s when the fun began. Because like most technologies, it turns out to be harder to make use of than you’d expect.

Let me start with some strategic background. Our product provides detailed behavioral analytics on what people do in physical spaces. Our most important vertical and our most complex is retail. The behavior of customers in stores is rich, interesting, important and potentially actionable. We measure it (anonymously using Lidar) in quite remarkable detail. We know every step someone takes in the store from the minute they enter to the time they leave. We know which displays they walked past. Where they lingered. How much time they spent. Whether they interacted with Associates. And whether they ended up buying anything. It's a lot of interesting data.

Our DM1 platform provides a bunch of different visualizations and reports of this data – everything from full playback of what happened in the space to traditional reports down to the 10-minute level of 50+ usage metrics, to charting, to funnel, path, and day-time part analytics. It’s all the usual stuff you’d see in any vertically focused BI tool with a layer of visualizations targeted specifically to our kind of data.

From a strategic perspective, we thought ChatGPT might solve three problems we see with our platform. Like many BI systems, the hardest part for most analysts isn’t getting data, it’s using it. We believed ChatGPT might function as a kind of analytics co-pilot (I’ll borrow the Microsoft lingo here) highlighting what’s interesting in the data you’re looking at inside our platform. The idea is that when you pull a report, ChatGPT will tell you what might be interesting to think about.

There are also times when people need to get at specific data. Like most enterprise systems, we provide a full feed for times when the data people need isn’t really available in the interface. We have, for example, a full Big-Query feed. Yet there are lots of potential data consumers who would like to get data but aren’t good enough at SQL to reliably access it. We figured ChatGPT’s SQL generation could be trivially integrated into the product to solve this problem.

Finally, the biggest challenge that most users face with analytics data is getting started using it. Even if you have specific questions, many people lack the skill to translate that question into a research strategy using the data. We believed ChatGPT might help by functioning as a smart starting point calling out key changes in the data, helping stakeholders find interesting and important changes, and perhaps replacing the entire edifice of data exploration with a simple Q&A interface.

Over the last year, we’ve built, tested and explored interfaces to ChatGPT for all three of these uses. None proved trivial. None were quite as successful as we’d originally hoped. But at least several have added real value.

Slapping AI on a product-label is ludicrously easy. And people are already getting fed up with products that advertise AI or ChatGPT but don’t work nearly as well as ChatGPT. I can tell you from personal experience that even when you integrate ChatGPT into an application, you often get something that works a lot worse than the original ChatGPT. In this series of posts, I’m going to walk through our actual experiences trying each of these directions. I’m not going to go much into the coding (which is mostly trivial), but I am going to talk about the tactics we used, the problems we ran into, and some of the solutions we eventually settled on.

I hope it’s useful learning for others who want to follow a similar direction and for anyone who can’t understand why ChatGPT seems so great and yet many of the resulting Chatbots and applications don’t seem nearly as good.