Applying LLMs to Data Analytics

One of the areas in Data I am most passionate about is ANALYTICS AS CODE. I am bullish about this because I believe this is where AI can deliver the most impact to the end users of Data Applications such as Business Intelligence. For me, the biggest revelation from the recent Google post on Gemma release was not what was in the tweet, but what was not: the role of applications.

When it comes to code, underlying models get you only to 20-40 points. The majority of the value (as much as 40-60 additional points according to my own company’s benchmarks) comes from the application-level fine-tuning.

Ok, let’s dive in…

1) Mode Analytics

Mode is an interesting vendor because it started as an internal project at Yammer, and did not have very impressive—or original—architecture to begin with. In some way, it committed the very mistake Mode’s Co-Founder claims are committed by the majority of Data start-ups: “internal (data) tools make lousy startups” (Benn Stancil).

And yet over time it has developed into a comprehensive Analytics tool - one that is capable of serving everyone from a Data Scientist, with their implementation of Python notebooks, to an executive - with fancy Dashboard functionality.

Looking at Mode’s AI Assistant, however, I have to conclude that it is non-impressive. Mode’s AI assistant works by taking the context of the current query and sending it to a raw non-fine-tuned OpenAI model. In this way, Mode’s assistant’s has intelligence neither about the data model of the business, such as column names or business-specific metric definitions, nor about the the domain of the business - such as domain-specific ways for marketing attribution. It is helpful only in the context of a very long query with multiple other Common Table Expressions (CTEs) - from which the assistant can extract some useful information. Basically, it is nothing more than a simple OpenAI extension.

2) Hex

First, it is worth crediting Hex with succeeding where many have failed: notebook collaboration. Between 2015 and 2020 this was a very busy category. VCs have invested into numerous start-ups (e.g. Deepnote, etc) that have largely failed to grow into unicorn-level companies that they pitched their products to be.

Hex works by merging a few concepts together: SQL, Python Notebooks, and “SQL Chaining”. Each new section of the notebook lets analysts apply a transformation to the previous result.

On the surface, Hex’s AI product, Magic, succeeds where others fail. Unlike Mode’s AI, Magic relies on all available context. Whether that’s the vendor’s knowledge of underlying database schema, or information available about partner integrations, Magic takes that meta information and sends it over along with the user prompt to OpenAI. It also has a backend Data Manager interface, where the customer can either provide additional description to the data model OR exclude certain data sources entirely from AI. See for yourself.

However, testing the AI deeper reveals major flaws. A simple SQL test (not even SQL101) to calculate the average price of items in the order fails. The AI does not recognize the difference between the price of an item from a menu table and that of an order from the orders table (a combined total of menu items). For all the increased context, Hex’s Magic is not actually that magical as it relies on the basic raw OpenAI model without adjusting it for any of the knowledge Hex team has about typical data use cases.

The other area Magic fails in my own testing is speed. Where the video demo succeeds, the real use of Magic—even in a very small test scenario ran from a trial account—takes forever to process. If they are sending all meta information in the context window without pre-fine-tuning, it could explain the major lag. The delay there has a relationship of O(n^2) between time and the amount of information in the context. Naturally, this is not a good way to do LLM implementation.

Basically, what Hex has done is put a Tesla battery into a Ford, but made no other significant changes, so the car continues to mainly rely on gasoline.

Basically, what Hex has done is put a Tesla battery into a Ford, but made no other significant changes, so the car continues to mainly rely on gasoline.

3) Looker

As a former Looker employee, I wanted to initially spend a lot of time delving into Looker (not to be confused with Looker Studio - a sliver of full functionality). In principle, Looker, being inside Google and all, has the most to benefit from AI. It has this incredibly powerful semantic layer that captures code cohesively across all analytics into one standard model.

With the flip of a button, all those models shared with Google could be trained on to build a new LLM. With nearly 10,000,000 business users and hundreds of thousands of data developers - this would essentially give Google an unprecedented access to a proprietary codebase to build the best AI for analytics, period.

Because the code from data teams is not readily available in the public, LLMs today are trained only code snippets that are readily available from documentation and places like StackOverflow - not representatively of the true complexity of code inside actual data teams.

But as I dived into the overall Gemini phenomenon and its Looker implementation, my enthusiasm has vanished.

Looker essentially has outsourced all AI to Gemini. Without any special access to LookML from Gemini’s underlying models for training purposes, the AI assistant adds little leverage over just someone connecting a Github copilot or OpenAI extension to their VSCode editor and making changes to LookML there. The implementation is basically analogous to that of Hex, but relies on Google rather than on Microsoft/OpenAI. It has no special understanding of Analytics (or LookML for that matter) other than what Google’s own servers were capable of scrapping from public web. And with Google’s recent Gamma launch flop, personally I would be tempted to keep my Looker code and AI copilot engine hosted on separate vendors - not all in Google.

That is it for now. Did you enjoy reading this? Then please comment bellow and let me know which other vendors I should cover.