Week of June 9th
Stefan Krawczyk
CEO @ DAGWorks Inc. | Co-creator of Hamilton & Burr | Pipelines & Agents: Data, Data Science, Machine Learning, & LLMs
TL;DR:
Hamilton Release 1.66.0
Highlights:
Hot on the heels of Databricks data & AI summit, we're pleased to announce some great support for MLflow .
MLFlow: MLflow is a project that is popular for storing things like model metrics and even model artifacts. With this new release you have:
What this means is that you don’t have to include / integrate MLFlow directly into your Hamilton code, instead you couple it together at the Driver level. This enables a clean separation between logic and runtime concerns.
from hamilton import driver
from hamilton.io.materialization import to
from hamilton.plugins.h_mlflow import MLFlowTracker
dr = (
driver.Builder()
.with_modules(model_training_2)
.with_adapters(MLFlowTracker()) # <- add this, it'll autolog to MLFlow
.with_materializers(
to.mlflow(
id="trained_model__mlflow",
dependencies=["trained_model"],
register_as="my_new_model",
),
)
.build()
)
For more details see this video overview and this tutorial notebook.
Hamilton UI Update:
Before you needed to have Docker installed to run the UI. Now you don’t!
then
to start it.
This should enable you to quickly and easily explore your Hamilton DAGs — just add the adapter to your driver (follow the instructions in the UI) and then it’ll log to it; you don’t need to execute it to be able to see it in the UI.
Fixes:
Examples / Documentation Updates:
WrenAI
We're excited that Hamilton is being picked up by another open source library. This time from Wren AI . They are building a RAG system and using Hamilton to help orchestrate it!
Blog: Lean Data Automation: A Principal Components Approach
领英推荐
This blog post was written in collaboration with Runhouse . In it we discuss that by unbundling the principle components of (macro) orchestrators, we can take advantage of a lean, cost-effective, and flexible stack. This can be done in such a way, i.e. by choosing the right tools, to preserve all the visibility, collaboration, and scale we need.
In the post we have a code example of this stack. It uses Github Actions, which is a free and widely available scheduler, and then combines using Hamilton with Runhouse, i.e. two open-source dedicated asset and infrastructure layers, to create this nimble and lean approach.
My main take away is that you can get pretty far before you have to reach for something like Airflow, Dagster, or Prefect, for data & ML work.
Blog: Traveling back in time with Burr
One of the best features of Burr is the ability to "fork state", i.e. given some application run and a point in time, copy that state into another application for you to debug/iterate with.
We wrote the hows and whys of this up in a post. That also comes along with a user contributed video on how they use this approach to develop with Burr! Thanks Ashis Ghosh !
The TL;DR: to enable it, is that you just need to pass in the write "IDs" to know where to take state from when building your application:
.initialize_from(
state_persister,
resume_at_next_action=True,
default_state={"count" : 0},
default_entrypoint="count",
fork_from_app_id=PARENT_APP_ID, # <--
fork_from_sequence_id=PARENT_APP_SEQUENCE_ID # <--
)
Burr in Python Weekly
Always fun to see our projects get picked up onto various lists.
New Burr example: using it to power an OpenAI compatible endpoint.
Thierry Jean came up with a cute idea. There are many UIs that allow you to interface with an OpenAI compatible endpoint easily. Wouldn't it be nice to use one of them to interact with your Burr application?
Well we now have an example that precisely shows this.
The idea is simple - expose a FastAPI endpoint that mirrors the OpenAI endpoint. Then underneath, it delegates logic to Burr, which in turn can do whatever you want!
Here's a video walkthrough of it.
Hamilton OS Meetup Group
June meet-up is this coming week! Sign up here
New functionality:
Co-creator of Hamilton/Burr OS libraries, Co-founder @ DAGWorks (YC W23, StartX S23)
9 个月Really can't believe we got all this done this week