Rill Data

Rill Cloud is now in public beta! Skip the data warehouse, and transform an S3 or GCS bucket into an interactive dashboard with just one tool. Rill's fast, exploratory dashboards are: ? shareable with team members & partners ? continuously integrated & deployed via Github ? always up-to-date with fresh data This is a big step towards making our vision of “data lake to dashboard in minutes” a reality for software engineers everywhere. https://lnkd.in/gAGwXFtX

Check out our powerful filtering capabilities now available in Rill 0.58! ?Contains filters - allows you to apply a true search filter on your data ?List filters - have a long list of Campaign IDs from a external system? No problem! Copy/paste them into Rill & quickly filter down your dashboard to that specific list.

A new blog post, in which I dust off my ETL and data modelling skills, and build a data pipeline with DuckDB (and try out some data viz tools including Rill Data, Metabase, and #ApacheSuperset): https://lnkd.in/e4xcXgPH #dataEngineering

Analytics & reporting data in podcasting is crucial, but at the same time inventory is split across numerous platforms making it hard to get the full picture. In 10 days, we'll be at Podcast Movement in Chicago. We’d love to show how we ingest data at any cadence & build fast dashboards.

Data peeps - We're hiring a Staff Data Engineer at Rill. If you or someone you know loves building complex data pipelines at massive scale, please DM me. This role is focused on orchestrating and transforming terabytes of data from lakehouses into real-time databases (ClickHouse, DuckDB, StarTree/Pinot, StarRocks), doing so with modern tooling (Dagster Labs, Astronomer, Tobiko), and optimizing data models for high performance within Rill's fast exploratory dashboards. https://lnkd.in/gVYReUnc

Text today from a founder: "Hey, consulting with an AI startup now. They have a bunch of operational data in Postgres and their Looker queries are super slow. They are looking to ETL to Redshift... is this a good idea?" My answer: "Yes, you need an analytical database... but maybe not Redshift." Redshift *is* a warehouse for analytics, just like BigQuery and Snowflake. These cloud data warehouses have an architecture -- namely the separation of compute and object storage, plus a columnar data layout -- that enables cost-efficient analytical queries like filters and aggregations. But this cost-efficiency has its own cost: performance. CDWs are fine for building reports, but running exploratory dashboards on these behemoths is a one-way ticket to spinner city ("??... loading....??"). This speed-at-scale gap has led to the surging popularity of real-time analytical databases like ClickHouse, Pinot (StarTree), and StarRocks. Unlike cloud warehouses, these databases co-locate compute and storage, then further leverage vectorization, parallelism, aggregation, and novel indexing structures to achieve sub-second queries on terabytes of data. Companies like Netflix, Spotify, Uber, Stripe, AirBnB, and OpenAI leverage these real-time analytical databases to power operational, exploratory dashboards about media sessions, trips, payments, reservations, and API calls across billions of events. These always-on digital platforms live and die by operational metrics, and investigations into these metrics require fast, flexible dashboards -- not slow, rigid reports. Real-time analytical databases also have an unexpected bonus: slashing costs! They shield warehouses from a class of frequent, full-scan, costly queries that dashboards generate. One of our recent customers at Rill cut their annual 7-figure Snowflake bill in half by offloading core metrics to a real-time analytical database. If your dashboards are slow and you're wondering if a real-time analytical database is right for you, data engineer and author Simon Sp?ti recently published a detailed guide about how to choose between the many different options. (Link to his blog post is in the comments). p.s. Even if you don't have scale, you may still need speed. DuckDB is an embeddable real-time analytical database that enables sub-second performance on a single node -- which these days can be pretty beefy (on Rill Cloud, we run DuckDB-powered dashboards at scales into the 10s of GBs). And the team at MotherDuck is building a cloud-scale version of DuckDB. p.p.s. Why do I care so much about analytical databases, besides being a nice guy who answers text messages my friends? It's because I co-founded Rill Data to be a fast BI tool built for these fast databases.

We love collaborating with Simon Sp?ti on the latest trends in data like declarative data stacks and GenBI!

A year ago, Michael Driscoll was on The Data Stack Show talking about fast OLAP engines and the impact on BI. This topic is still very relevant today. If you missed the podcast before, check it out now to see how things have progressed ?? https://lnkd.in/gSbiUxwa

While pivot tables are great, sometimes you just need a simple, straightforward table of data. Starting today in the latest Rill .57 release, you can toggle between nested and flat modes for your table, making it easy to switch between a pivoted view and a flat view. In flat mode, you can freely arrange measures and dimensions interchangeably, giving you more flexibility in organizing your data.

Cloud data warehouses like Snowflake use decoupled cloud storage + compute architectures, offering lower cost but with lower performance. Conversely, real-time databases like ClickHouse, Pinot, Druid, and DuckDB achieve higher performance by co-locating compute and storage. The significant difference between these analytical databases is the trade-off between cost (per-TB-stored) and performance (per-TB-scanned). Choosing the approach and solutions to replace a cloud data warehouse for low-latency analytics solutions requires careful consideration of specific needs, constraints, and objectives.