DBOS, Inc.

Earlier this month, I delivered a talk on "Time Travel Debugging with Postgres" for the Seattle Postgres User Group. Massive thanks to Jeremy Schneider for recording (and editing) the talk and posting it to YouTube! (Note to self, get the details of Jeremy's multi-camera + Google Meet for screen capture setup. That was top notch.) I really appreciated the extended Q&A that we had after the talk. The video below is just over 90 minutes, but only half of that was my talk and demo. The other half was all Q&A, hitting wide ranging topics regarding durable computing, query parsing, writing the worlds ugliest switch statement and cross compiling C code for Windows ARM64 using Zig. That's my kind of Q&A. In the talk, I mentioned that the time travel demo was running on a hacked up...err "locally modified" version of DBOS Transact to work around a bug I had found the day before. Conveniently enough, DBOS, Inc. shipped v2.6 *today* that includes the fix for that bug. Additionally, since this video was recorded, we've shipped Replay Debugger support in the DBOS Debugger for VS Code. In other words, everything you see in the video you can try yourself at home today. https://lnkd.in/gmv4KRFV

Why does it take so long to load data from your database? This paper tackles the systems challenges underlying a common data science problem: loading large amounts of data from a database into an in-memory dataframe for data pipelines, ML inference, or exploratory data analysis. What I like the most about the paper is its opening: an in-depth analysis of a popular data loading operation (Pandas.read_sql) and why it takes so long to load data. The way this operation works is simple: it sends a SQL query to a database to request some data, deserializes it into Python objects, then converts them into NumPy arrays backing a Pandas dataframe. Somewhat surprisingly (or not, depending on how cynical you are about Python performance), more than 90% of the load time is spent client-side, deserializing and converting the data after it has been fetched from the server. Moreover, these operations use 4x more memory than the data actually requires. This happens because the underlying implementation is naive: it fetches all the data into an in-memory buffer, then converts it all into Python objects, then converts it all into a dataframe, with no parallelization of any step. Not content with complaining about the problem, the authors propose a solution: ConnectorX, an improved data loading library which is open-source and available on GitHub. The main idea is to stream data from database to dataframe. Before transferring anything, ConnectorX queries the DBMS for metadata about the query result, and then pre-allocates the entire dataframe. It then partitions the data client-side and assigns each partition to a separate worker-thread. Each worker thread streams data directly from the database into the dataframe, fetching small batches of query results from the DBMS, converting them into the proper format, then writing them to the dataframe. To make things really fast, all of this is written in Rust with Python bindings. Overall, this combination of a faster language, more parallelism, and a pipelined streaming approach reduces client-side overhead from 10x to almost nothing.

I recently got asked an interesting question about separation of concerns in durable execution. The conversation was about whether it is better to add durable execution to an application using an external orchestrator like Temporal or a library like DBOS. The specific question was whether external orchestration was better because it separates concerns, allowing the application and its orchestrator to be managed separately. The answer I gave is that this is actually a disadvantage of external orchestration because those concerns aren’t meant to be separated. In reality, the orchestration/workflow code in an app is also the app’s core control flow. For example, in a data ingestion pipeline that extracts and transforms data through several steps, the actual pipeline code that orchestrates the steps is as critical as the implementations of each step. Typically, the orchestration code is designed and operated by the same team that builds the app because it IS the app. What I've heard from talking to developers is that separating that control flow into an external orchestration layer adds friction to every step of the development process because their app is now distributed across multiple systems. Essentially, you’re decomposing your own application into multiple microservices–one that orchestrates the workflow and another that executes the steps (and perhaps a third that acts as a client), with all the complexity that entails. By contrast, a library like DBOS only requires you to add annotations to your application to obtain durable execution, no rearchitecting required.

Want to attach a debugger to that weird error you got in production yesterday? With the latest release of the DBOS debugger, you can get pretty close!? It lets you take any past trace of a DBOS workflow, execute it locally, and attach a debugger so you can step through it and see what really happened. Under the hood, this works because DBOS checkpoints the execution state of your workflows (what steps have completed and what their outputs were) to Postgres. It does this for reliability, using the checkpoints to resume interrupted workflows from their last completed steps. However, this saved state also enables replay debugging! The debugger uses the checkpointed information to re-execute the workflow, reconstructing its state at each step of execution. Then, you can single-step through the workflow and see exactly what it did (what the values of local variables were, what branch of execution was taken, whatever you like). More info below: ??