DAGWorks Inc.

hey hey - here's what's been cooking with us this past month. #Hamilton highlights: ?? crossed 2000 github stars ?? > released multithreading based DAG parallelism Want to easily parallelize a DAG doing lots of LLM calls? Easy. Try our multithreading based approach. > RichProgressBar adapter Want a better cmdline experience? Check out this adapter to help tell you where Hamilton is with respect to computation. Thanks to Charles S. for the contribution. > Hamilton installs in a WASM environment. We're excited to enable more WASM based use of Hamilton with tools like marimo. We see more future use of python in the browser and we're excited to support using Hamilton in that context. #Burr highlights: ?? crossed 1500 github stars ?? > async persisters We added asynchronous lifecycle hooks and implementations to enable more performant persisters when using asynchronous python with Burr. Thanks to Jernej Frank for the contributions here! > tag aliases for actions You can now tag all actions that require input, or responses, etc. and then use that tag when running your Burr application. As graphs get larger this is one way to more easily manage your control flow logic without having to update it each time your actions change. > Burr installs in a WASM environment Note: Burr has always worked in a WASM environment, so no new features here. For more details, jump into the newsletter.

Fun collaboration with ScrapeGraphAI! In this post we talk about building a simple web-scraper/chat engine with #Burr, LanceDB, and #scrapegraph. We leverage the scrapegraph sdk, but it can all be done with OS + free tools. https://lnkd.in/gyACTJ9c

Happy New Year! Here's a recap of the last month and some of 2024. Thanks to the community and our contributors this year. #Hamilton + #Burr 2024 stats: > 35M+ telemetry events (10x growth), > 100K+ unique IPs (10x growth) from 1000+ companies, > 1M+ total downloads. #Hamilton December release highlights: > TypedDict support, Snowflake example #Burr December release highlights: > Anyscale's Ray executor for parallelism > Persistence, i.e. fault tolerance, for parallelism Blogs: > We have four this past month! They cover Hamilton internals, parallelism, and TDD LLM application development! Links in the newsletter! In the wild: > Burr called out by AI Agent Ops Alliance? (AOA) > Jernej Frank's recording on the internals of Hamilton decorators > how to deploy the Hamilton UI on Snowflake

?? Happy new year! ?? We wrote a fun post on building parallel agents with #burr and #ray to get the year started. https://lnkd.in/grbTQnqv The hard problems at the center of it. How do you handle parallel tasks in your agent's workflow? What if the orchestrator dies/loses connection? What if the tasks are flaky? Burr provides a host of tools to solve these without much additional effort -- most of this just *works* out of the box. The idea is that spawned sub-applications inherit their parent's persistence capabilities, enabling you to restart any failed applications where you left off, and restart the entire suite of applications from their prior state if the parent dies, affording you an eventually completing workflow! We use: ?? #Burr to orchestrate/persist ?? #Ray by Anyscale to distribute out execution ?? Any db you want to persist data between runs The distribution engine and database we use are pluggable -- while the Burr library provides a few plugins, you can (and should!) easily write your own.

?? The choice of Agent Ops tooling depends on the specific use case needs and preferences. ?? For state machine modeling & implementation: --------- 1?? If precise control over state and detailed visualization, DAGWorks Inc. Burr might be the best choice. 2?? If you prefer a higher-level abstraction and seamless integration with LangChain, LangGraph could be a good fit. 3?? And if you need a more versatile tool for various NLP tasks, including agent development, Haystack might be the way to go. -------- ??? Burr: ??Explicit State Machine Focus: Burr is built with state machines as the core concept. You define "actions" as Python functions that explicitly read from and write to a central state object.? ??Fine-grained Control: Burr gives you very granular control over state transitions and actions, allowing for complex logic and conditional branching. ??Visualization: Burr's telemetry UI provides a clear visual representation of the state machine, making it easy to understand and debug agent behavior. ---- ??? LangGraph: ??Graph-based Representation: LangGraph uses a graph structure to define the state machine, where nodes represent actions or tools, and edges represent transitions.? ??Higher-level Abstraction: LangGraph provides a higher level of abstraction compared to Burr, focusing on the flow of actions and data between different components. ??Integration with LangChain: LangGraph is tightly integrated with LangChain, making it easy to incorporate various LLM chains and tools into your agent workflow. ---- ??? Haystack: ??Pipelines with State Machine Elements: Haystack's Pipelines feature incorporates some state machine concepts, allowing for conditional routing and branching within a workflow.? ??Less Explicit Focus: However, Haystack doesn't explicitly enforce a state machine structure for all applications. It's more flexible and can be used for a wider range of NLP tasks beyond agent development. ??Limited Visualization: Haystack doesn't have a dedicated UI for visualizing state machines like Burr. -------- ? Agentic Systems are the future of AI - AI Agent Ops Framework? (AOF) Unlocks the Potential ?? Join the industries' only dedicated AI Agent Ops Community: https://lnkd.in/dMDFZMJa

I'm really excited about some new capabilities we launched in #Burr! You can now easily run parallel, multi-agent workflows with a set of abstractions that enables a simple map/reduce pattern -- E.G. generating multiple candidates to evaluate, running a variety of simultaneous tools, scraping multiple web-pages, etc... We wrote about it here -- I'm happy with how Burr has grown to enable more and more powerful agentic systems:

Really excited for this guest blog post by Jernej Frank. This is something new -- he's one of the #OS contributors that dove deepest into #Hamilton, and has had a massive impact on some of the core aspects including graph execution and lifecycle adapters. In this post, he digs into an abstraction that we designed to make building ergonomic DAGs easier -- how Hamilton leverages python decorators to enable modular, readable, and extensible data transforms. Worth a read -- there's a bit for everyone! He goes over the basics of Hamilton, the trade-off between explicit/verbose code, how hamilton leverages decorators to mitigate that trade-off, and the architecture details of how that works. On a personal note -- I've also really enjoyed working with Jernej Frank as he dove into the deep end of Hamilton's implementation. Goes to show how #opensource can build and a community and bring a ton of folks along for the ride! https://lnkd.in/gZfZXCs8

Tired of the deployment headaches with #LLM agents? BentoML takes care of the heavy lifting - from REST endpoints to scaling. Great post showing how to combine #Burr and #BentoML for streamlined agent deployment. Read the blog post to learn more: https://lnkd.in/gJNccqw9

Excited to see #burr / DAGWorks Inc. mentioned here!

Really excited to talk about building out #AIAgents reliably with #Burr at PyData Global next week! https://lnkd.in/gafKvBM9 If you haven't already, make sure to register -- great roster of speakers!