Arroyo

If you missed #p99conf last week, talks are now available to stream on YouTube. I spoke about the design decisions that went into Arroyo's incredible performance: https://lnkd.in/g8-rrGWR. Come for the Rust hot takes, stay for my terrible hand-drawn architecture diagrams ??

We've been able to build a great open source community around Arroyo, with outside contributors adding major features and improvements—even though it's a streaming SQL engine, a piece of deep infrastructure with a high barrier to entry. Building a real community is something lots of projects struggle with. How did we do it? ? Starting with a friendly community meeting place where new contributors can meet the team, ask questions, and find mentorship (for us this is Discord) ? Doing the work of creating (and tagging) issues specifically for new contributors. This takes a lot of effort! They need to be well-documented, with enough context for someone to pick up cold. ? Cleaving off a part of the codebase that's mostly disconnected, with clean integration points to the rest of the system. For us this is our connectors subproject. which contains code to connect Arroyo with other systems. We've had multiple big contributions here, including NATS and MQTT connectors. ? Providing efficient PR reviews and actively helping users get their changes merged. Nothing kills motivation like waiting 2 months for a review. This all takes work and time, but we've found it incredibly worthwhile. (And if you've ever been interested in contributing to an open source data infra project, get in touch!)

The Arroyo team is thrilled to announce that Arroyo 0.12.0 is now available! This release introduces ?? Python UDFs ??, which allow developers to extend the engine with custom functions. Also new in this release: ?? Support for Protobuf as an ingestion format ?? Much faster JSON functions and new PG-inspired JSON syntax ?? Custom TTLs for updating state ?? AWS IRSA support along with many other improvements and fixes. This release wouldn't have been possible without all of our amazing contributors, including several new to the project: ? Xin Hao (@haoxins) ? Jayshan Raghunandan (@jr200) (new!) ? Marco Lugo (new!) ? Micah Wylde ? Tiago Campos (new!) ? ZhuLiquan (@zhuliquan) (new!) With Python support, we're excited to bring powerful stream processing to a whole new set of developers. We can't wait to see what you build! https://lnkd.in/g-dEyBqh

Excited for the SF DataFusion meetup next Wednesday! I'll be giving a talk about how Arroyo implements dynamically-loaded UDFs. Because Rust lacks a stable ABI, this is harder than it sounds—different compiler versions or even changes to flags can break code loading. But we don't want to recompile our entire engine just to use a UDF. This gets even harder if we're trying to use async across a UDF boundary (which Arroyo has to support to enable things like HTTP calls, database lookups, and model inference in UDFs). How do we do it? You'll have to come to the meetup to find out. But I'll give you a hint: it involves C ?? See you there!

Arroyo is coming to Current 2024! Excited to see everyone in Austin next week.

Recently one of our users in finance asked if Arroyo (YC W23) could write to InfluxData. We don't have an InfluxDB sink, but I thought it might be possible to build a good integration using the Influx HTTP API and the Arroyo webhook sink. Turns out... this works great! Amazing to see how a few different features + standard SQL compose so well to power use cases we haven't designed for.

Stream processing pipelines are hard—hard to build, hard to deploy, and hard to operate. Right? What if I told you it was possible to deploy a stateful streaming pipeline to the serverless cloud platform Fly.io? Spoiler: with Arroyo (YC W23) you can! And it's so lightweight it runs within the Fly free tier. At Arroyo our north star is to bring stream processing to every company. But until now, deploying Arroyo still required a distributed cluster and complex infra like Kubernetes. In 0.11 that changed, with the introduction of a deployment mode we call?pipeline clusters. Pipeline clusters bundle up the control plane, data plane, and query logic into a single, easily deployed unit. And that makes them a great fit for deploying streaming pipelines on serverless clouds. How is that possible? How can we run a?stateful streaming pipeline?on a serverless platform meant for stateless microservices? The secret ingredient is an?object store. By checkpointing our state (for example the in-progress windowed data we're aggregating over) and our config database (sqlite) to an object store like S3, we're able to run consistent, stateful, correct streaming pipelines on unreliable, easily managed processing nodes. Fly.io has long been our favorite cloud provider for webapps, and with their recent support for Tigris Data object storage it's become a great place to run data applications like Arroyo. So I wanted to see how quickly I could get a reliable, fault-tolerant, stateful streaming application up and running on Fly. Watch the video to see how it went:

Our second community office hour is happening next Wednesday! The team will present our roadmap for the next set of releases and then open the floor. Come with your Arroyo war stories, questions, favorite bugs, and feature requests!

Announcing homebrew support! Arroyo is a complete SQL streaming engine in a single binary, and now it's just one command to get it locally on MacOS: $ brew install arroyosystems/tap/arroyo $ arroyo cluster Get started in 5 minutes: https://lnkd.in/gqDeBQzm

I had a great time talking with Micah Wylde, founder of Arroyo (YC W23), and walking through connecting WarpStream to Arroyo so that I could perform SQL functions on live data streams and then create pipelines with it. There is a lot of streaming data out there, and working on it without having to land it, as with the WarpStream serverless technology, is an incredible advantage. Shout out to ShadowTraffic for making the data simulation part of it simple. #apachekafka #kafka #dataengineering #datastreaming #streamingdata. https://lnkd.in/guicDnNh