User-Facing Analytics Supercycle and My Reasons to Join StarTree

Data is a perishable asset and experiences exponential decay in value.  However, real-time data analytics has largely remained an unfulfilled wish for many companies primarily due to the technology gaps in modern data architecture and has prevented them from maximizing their growth potential.  LinkedIn and Uber pioneered real-time analytics to power user-facing data apps and realized tremendous growth as a result.  In the last two-three years, many open source technologies such as Apache Pinot are pioneering a new era of real-time analytics and the broader industry is taking notice.  In this blog, I will share my thoughts on 

• why real-time analytics to power user-facing data apps is now attainable, 
• how companies can take advantage of this shift to grow their core businesses, and 
• what makes StarTree truly an exceptional company.

I have had the good fortune of building and commercializing data infra and analytics products starting with SAP Business Analytics suite including in-memory columnar database called Hana and continuing this journey with Facebook (Meta), Netflix, and Moveworks.  

SAP Hana in its first year of launch captured a billion in revenue, a record in the on-premise enterprise market for data analytics, and disrupted pioneers including Teradata, Netezza, Exadata, and its own Business Warehouse.  However, specialized hardware-based analytics could only scale vertically (more memory, more CPU), and started to lose market share to distributed data architecture which could scale horizontally on commodity hardware.  The data analytics market has grown at least 100x since as data volumes continue to explode and companies looking to gain an edge have invested heavily in data infrastructure. 

At both Meta and Netflix, companies with arguably the most sophisticated data infrastructure, many transformative distributed data technologies were pioneered including Hive and Presto, both open sourced, to take advantage of the commodity hardware and deliver horizontal scaling.  These and many such companies including LinkedIn and Uber pioneered many modern data architecture concepts such as data lake, distributed data warehouse, cloud-based data visualization, and cloud native analytic apps to massively grow their businesses.  When the broader industry took note of the unparalleled advantages these companies had gained from their data architecture, it triggered a supercycle of investments in the development of the modern data cloud. 

As data workloads moved to cloud to capture new growth, the need for user-facing analytic applications, a new category, quickly emerged. The business value of data decays exponentially with time, a concept known as time-value of data.  No other category of companies leverage time-value of data better than the three-sided marketplaces including Wix, eBay, Shopify, Spotify, Etsy, Uber Eats, PayPal, Doordash, and many more.  These companies have developed and commercialized real-time user-facing analytics to build defensible differentiators to grow their businesses.  

For user-facing analytics applications, ultra low-latency, high concurrency, and data freshness are the three most important principles.  However, the modern cloud data architecture didn’t have the equivalent of an OLAP database which supported high data ingestion rates to power low-latency high-QPS applications.  As a result, many user-facing analytics applications were built on top of cloud data warehouses, RDS, NoSQL stores, and search stores leading to latency, data freshness, and throughput issues.  I have personally experienced sub-par performance of many data-science applications built on cloud data warehouse and other data stores.  Fortunately, in the past few years, many technologies and companies have surfaced to address cloud-native OLAP gap in the data cloud architecture.  Apache Pinot, built, productized, and hardened at LinkedIn and Uber, is leading this new cohort of cloud-native OLAP data technologies. 

Apache Pinot is a real-time distributed OLAP system that provides low-latency and high-throughput querying over PB scale datasets. It was designed for real-time analytics use cases that require sub 100 milliseconds query response times and very high real-time data ingestion.  At LinkedIn, Pinot is powering more than 200+ data applications.  Apache Pinot is a matured and established technology in the cloud-native real-time analytics category.  

StarTree, the company supporting Apache Pinot community, is relentlessly focused on driving the innovation in the field of real-time user-facing analytics. StarTree has developed a large number of defensible differentiators including indexes to supercharge query processing, upserts to support data mutability, tiered storage to reduce cost of queries over large amounts of data at low latency, large number of join types and is gaining parity with ANSI SQL.  StarTree also pioneered BYOC (bring your own cloud) to complement its SaaS offering to help customers take advantage of Pinot in their own cloud to supercharge user-facing analytics without much rearchitecting. 

I have worked in data infra and analytics for more than a decade and I am more excited than I have ever been to join a company which has clarity in its purpose, believes in developing a product that truly delights its users and raises the bar for customer success, and has assembled a talented group of people who are simply relentless. I found these three key forces — Purpose, Product, and People — operating in harmony at StarTree to potentially push this company towards greatness.

StarTree’s purpose to supercharge user-facing analytics to help unlock more growth for its customers is both clear and ambitious. The strength of the StarTree product lies in the power of the Apache Pinot open source community, commitment from its customers, and the pursuit of engineering excellence from employees. I truly believe that StarTree is a once-in-a-generation opportunity for its investors and employees alike.  I am genuinely humbled to be part of the StarTree mission and excited to join this brilliant group of people in building and scaling the best cloud-native distributed OLAP product in the industry.

StarTree is set up to win because they are the best at solving their customer’s real-time analytic problems, truly care about the growth and success of the Pinot OSS community, and have invested in building an open and transparent people culture to fuel a virtuous cycle of impactful innovation.  Head over to the StarTree blog to learn more about how our distributed OLAP product is reimagining user-facing analytics or check out the careers page to join us on this incredible journey.