Apache Pinot, an open-source project, progresses as StarTree enhances real-time analytics and observability.

At the Real-Time Analytics Summit held today, StarTree unveiled significant updates to its product lineup, aiming to enhance accessibility to large-scale, real-time data analytics and observability in the cloud era.

StarTree, a leading commercial vendor, is the driving force behind the Apache Pinot real-time analytics data store platform. Spearheaded by CEO Kishore Gopalakrishna, formerly an engineer at LinkedIn where the project originated, Pinot aims to deliver a dependable, high-speed data store equipped with the necessary indexes and optimizations for scalable real-time analytics. StarTree extends Pinot's capabilities by offering a cloud-based real-time analytics platform as a service. Notable enterprises such as Stripe, Walmart, and DoorDash rely on Pinot for their analytics needs. StarTree and Apache Pinot face competition from various technologies, including other open-source alternatives such as the StarRocks online analytical processing (OLAP) database.

Developing Reliable AI: Microsoft's Approach to Secure and Scalable Generative Artificial Intelligence

The primary focus of the Real-Time Analytics Summit revolves around two main aspects: the open-source Apache Pinot project and StarTree's commercial solutions that enhance and expand Pinot's real-time analytics functionalities. Major updates entail the introduction of a serverless cloud service, seamless integrations with prominent data visualization tools like Grafana and Tableau, the official launch of the ThirdEye observability service, support for vector search, and the introduction of a new cloud write API.

Real-time data serves various purposes, and with these latest enhancements, StarTree is intensifying its efforts to support users in achieving heightened observability.

ThirdEye introduces real-time analytics for business metrics.

In a bid to bolster observability use cases, StarTree unveils its ThirdEye technology today, now available as a fully operational service. Chinmay Soman, StarTree's head of product, explained to VentureBeat that ThirdEye serves the dual purpose of identifying real-time anomalies and facilitating triaging and root cause analysis. The emphasis with ThirdEye lies on handling intricate business metrics that conventional monitoring systems find challenging. Leveraging Apache Pinot as its foundation, the ThirdEye service is capable of computing metrics from data points in real time.

It's noteworthy that business metrics differ from system metrics pertaining to IT systems. For instance, Soman highlighted how ride-sharing giant Uber previously relied on PagerDuty to monitor all system metrics. However, PagerDuty struggled to monitor metrics related to driver supply, which are derived and computed metrics posing challenges for traditional monitoring systems.

New vectors are introduced for real-time analytics in Apache Pinot.

The latest Apache Pinot 1.1 release takes center stage at the conference, with a major highlight being the introduction of vector index support. Vector indexes have gained increasing importance and popularity, particularly in applications involving large language models (LLM) and generative AI. In recent times, many database technologies have incorporated support for vectors, with Google even announcing that all its cloud databases would now support vectors due to their fundamental significance.

Enabling vector index search in a database can be achieved through various methods. Apache Pinot 1.1 introduces support for Hierarchical Navigable Small Worlds (HNSW) graphs as one of the ways to enable this capability.

The introduction of vector index support in Apache Pinot's 1.1 release signifies a significant step forward for real-time analytics and database technologies. This advancement holds the potential to enhance performance, enabling faster and more efficient processing of large datasets. With the growing importance of large language models (LLMs) and generative AI, the ability to handle vector data efficiently becomes increasingly crucial. Moreover, the endorsement of vector support by major players like Google suggests a trend towards standardization in the industry, which could lead to greater interoperability between different database systems. Overall, the future of vector index support in real-time analytics promises greater innovation, expanded applications, and improved capabilities for data processing and analysis across various industries.