Databricks SQL AI Functions: How Data Analysts Can Easily Unlock the Power of AI?

One of the key challenges facing data analysts today is the increasing demand to integrate machine learning (ML) into their workflows. Despite this, many analysts lack the technical resources or expertise to seamlessly apply ML techniques in their day-to-day analysis. Earlier this year, Databricks conducted a survey of 150 customers. According to their feedback, a common pain point is the difficulty in accessing ML tools. While analysts crave deeper insights, they often find themselves bogged down by complex processes, such as moving data between systems or custom coding. They need easy-to-use ML tools.

Databricks addresses this challenge with Databricks SQL AI Functions. By bridging the gap between analysts and AI, Databricks SQL AI Functions enable any analyst with SQL knowledge to easily access and utilize advanced machine learning models.

• GOAL: to simplify AI adoption and deliver results quickly — without the need for third-party tools or custom code.

In this article, we will get into the needs of Data Analysts for integrating ML into their workflows and how Databricks SQL helps them address these needs.

The Customer Challenge: Making ML Accessible for Analysts

In the customer survey, Databricks product managers notice a common theme emerged.

That is:

• NEED: Analysts want to leverage ML models for better insights and more streamlined workflows, but they often face hurdles.
• CHALLENGE: The traditional workflow typically involves taking data out of SQL environments.
• PAIN: This requires working with Python, and then building custom rate-limiting mechanisms and infrastructure. This not only complicates the process but also makes it less efficient.

The question Databricks aimed to answer was:

• MISSION (Databricks): How do we enable analysts to harness AI directly in their SQL workflows?
• SOLUTION: This led to the creation of DB SQL AI Functions.

Introducing DB SQL AI Functions: AI for Analysts, Made Easy

DB SQL AI Functions is designed to democratize access to AI. It facilitates analysts ready-to-use ML models that are easily accessible through simple SQL syntax (a language familiar to Data Analysts understand). With a focus on providing a fully managed, out-of-the-box solution, Databricks’ team* of ML engineers and data scientists has ensured that the AI models require no infrastructure setup from users.

• RESULT: Data Analysts can access state-of-the-art AI models via easy-to-use SQL syntax. No need for working with Python code.

*The same team behind the Mosaic AI platform

Here’s a breakdown of the key features:

• Ready-to-use models in SQL syntax: No need for custom Python coding or external integrations. If you know SQL, you can now harness AI functions with ease.
• State-of-the-art ML models: From text classification to sentiment analysis and large language models (LLMs) like GPT-4, Databricks SQL AI Functions offers a variety of models for diverse use cases.
• Fully managed infrastructure: Analysts no longer need to worry about handling infrastructure issues like rate limiting. The platform takes care of everything.

A Real-World Scenario: Sentiment Analysis in Seconds

To demonstrate the simplicity of Databricks SQL AI Functions, let’s take a common task:

Analyzing tweet sentiment.

By leveraging the AI analyze sentiment function, a Data Analyst can drop in tweet data and, within seconds, get a sentiment analysis: positive, negative, or neutral. The power of this function lies not only in its speed but also in how easy it is to use with minimal lines of code.

Furthermore, by combining other AI functions like AI classify, data analysts can quickly categorize data, creating actionable insights from unstructured datasets. Using just four lines of SQL code, an analyst can classify tweets and visualize the results using pivot tables — all within seconds.

Announcements: Faster, Smarter, and More Capabilities

In response to customer feedback, Databricks is continually enhancing the capabilities of Databricks SQL AI Functions. Their latest updates include:

• 10x performance boost: Querying custom models and LLMs is now 10x faster, thanks to optimizations like parallelization, batch APIs, and multithreading.
• New time series forecasting and vector search: These functions bring advanced forecasting and data retrieval capabilities to analysts, further expanding the range of use cases Databricks SQL AI Functions can support.
• Vector search with Mosaic AI: By leveraging state-of-the-art KNN search across data indexes, analysts can now easily perform rapid and large-scale vector searches on datasets of up to 100 million records.

What’s Next?

Databricks is committed to adding more AI functions, enhancing the speed and efficiency of existing ones, and helping data analysts unlock new possibilities. Whether it’s improving customer engagement, identifying trends, or forecasting future outcomes, Databricks SQL AI Functions is paving the way for a new era of analyst-driven AI.

By simplifying ML integration, Databricks empowers analysts to go beyond traditional data analysis and truly transform their insights — one SQL query at a time.

Conclusion

DB SQL AI Functions represents a breakthrough in how AI can be adopted by data analysts without requiring a deep understanding of machine learning or complex infrastructure. With faster performance, enhanced capabilities, and out-of-the-box models, it’s easier than ever for you to unlock the full potential of AI in analytics workflows.

This article was inspired by the talk AI Functions: Using AI in SQL at Databricks Summit by Jeremy lewallen Product Manager, Databricks.

You can read the full article by IntellaNOVA here:

Databricks SQL AI Functions: How Data Analysts Can Easily Unlock the Power of AI? | by IntellaNOVA | Oct, 2024 | Medium

Ali Ghodsi Ghodsi, CEO and Co-Founder of Databricks.

Matei Zaharia , Co-Founder and Chief Technologist (also one of the original creators of Apache Spark).

Jeremy L. Proformance PM, Databricks

Databricks #databricks #machinelearning #SQL #DatabricksSQL #SQLquery #dataanalysts #mlmodel #AI #ML