Dagster Labs

Just posted a GitHub discussion re: a preview of new @dagster features we are working on. We believe they'll be transformative: A new cli; improved project layout with Ruby-On-Rails-style scaffolding; and an opt-in, customizable YAML frontend: https://lnkd.in/e-E2F2zG

Are you struggling to choose the right LLM for your AI applications? Our latest blog shows how to implement intelligent model routing to: ?? Improve accuracy by up to 25% ?? Cut costs & latency by 10x ? Automatically select the optimal model for each query Learn how to create a "meta-model" that knows when to use cost-effective models vs. powerful ones, with practical Dagster pipeline examples using Not Diamond's routing capabilities. Check out the full tutorial with code examples #AI #LLM #Dagster #NotDiamond #MachineLearning

During a pair programming session to close up some open PRs on our Dagster repo, my colleague Luciano Antonelli told me he was surprised to find out Dagster Labs only has around 50-ish employees. I adamantly responded that he’s obviously making a mistake and this just can’t possibly be true given the quality of their software and platform. It took me a couple of Google searches and multiple fact checks to get convinced that he was indeed correct I was the one making a mistake! Deep inside I still feel like someone must have tweaked those numbers, because it’s just impossible to deliver such an impressive and robust platform with a team of this size! [Hmm… Unless they actually use Dagster to orchestrate their workflows?! ??] If you are not familiar with Dagster, do yourself a favor and watch some demos, videos, and maybe even get your hands dirty with a small pipeline to see what makes it so special. If you’re like me, at first you may not fully understand how their asset-based (vs task-based) paradigm is changing the game for data orchestration. But once you do, you won’t be able to go back. And you keep asking yourself why isn’t this the default way to build data pipelines in the first place?! If in 2025 you are picking Airflow over Dagster for your data pipelines, you’ve completely missed the point of what Dagster is bringing to the proverbial Table (pun intended!). It’s not just another orchestration tool. This is some mind blowing magic from the future that we get to use today, so when s**t breaks loose, it doesn’t spread all over the place in your warehouse. It stays well-contained where it happened, so you just jump in there and wipe it away in one fell swoop! (sorry, I’m sure there was a better way to describe this, but I’ve been changing too many diapers lately). Thanks Dagster Labs for bringing us a piece of the future! #respect #data #dataengineering #datascience #machinelearning #datapipeline

You heard it here first - we're hiring our first Data Engineer at Bookboost to take our data platform to the next level. ?? If working with the latest and most innovative tech (eg. Dagster Labs, dbt Labs, Snowflake), building an industry-defining Customer Data Platform and working autonomously (100% remotely or within our office hubs in Malmo or Amsterdam might I add) sounds interesting to you, then please reach out or apply directly via our Careers page (link in comments). Note: recruiters/agencies, we are currently filling this role internally so don't take my cold shoulder personally. ?? #datajobs #hiring #careers

Dagster has pioneered the so-called "Meatball Sub Architecture" for Code Locations. Like juicy meatballs separated by a layer of provolone and red sauce, you can isolate and organize modules by team, client, or workstream to be more and gain the benefits of high-level observability and operational control for your data platform. Code Locations enhance: ??? Organizational Clarity ?? Fault Tolerance ??? Isolation and Dependency Management Read more about code locations today! Link in the comments #dataengineering #dataplatforms #codelocations #meatballsubs

Want to break into Data Engineering in 2025? The traditional path into data engineering is broken. With 900+ applicants for a single position (and 90+ with referrals), how can anyone break in today? Most entry-level candidates are still focusing on the wrong things. While everyone chases the shiny new tools, they're missing the fundamentals. If you're early in your career and looking to transition to DE, here's what actually works in 2025: 1. Master the core technical skills first - Python, SQL, and Spark. These are table stakes, not differentiators. Too many candidates list these but fail technical screens. 2. Build real pipelines with cloud services. AWS, Databricks, and Snowflake all offer free tiers. Create projects that solve actual problems, not just tutorials. 3. Focus on "boring" industries. Civil service, banking, and insurance companies are desperate for talent while everyone chases FAANG salaries. 4. Understand data engineering is more than code. The best DEs understand the business context and can translate requirements into solutions. This is where analysts actually have an advantage. 5. Network strategically. Referrals aren't just helpful - they're practically mandatory now. Find communities where hiring managers hang out. What I've noticed is that candidates from analyst backgrounds often make better DEs than pure software engineers. Why? They understand the data itself, not just the pipelines. The market is tough, but opportunities exist for those who approach it strategically. #DataEngineering #CareerAdvice #DataPipelines #CloudComputing #TechCareers

Do you ever notice how job descriptions seem to demand experience with tools you've never touched? I was speaking with a data engineer recently who shared their frustration: "Every job posting wants Snowflake, dbt, or Databricks experience. How am I supposed to get that experience without already having a job that uses them?" This chicken-and-egg problem is real, but there are practical ways to break through. First, understand that many of these tools offer free learning paths. Snowflake University provides excellent hands-on training with badges you can showcase. The free trial gives you $400 in credits - more than enough to complete multiple badges and build sample projects. For dbt, you can download dbt Core (open source) and connect it to a local database like DuckDB. Build a small transformation project and push it to GitHub. This demonstrates not just tool familiarity but engineering best practices. Databricks offers a Community Edition that lets you experiment with Spark notebooks and Delta Lake. While it lacks some enterprise features, it's perfect for learning the core concepts. Employers truly value not only tool knowledge but also your ability to apply data engineering principles. SQL skills, data modeling expertise, and understanding of distributed processing concepts are transferable across platforms. Build a small end-to-end project using these tools. Document your learning journey. Being able to resolve blockers on these projects is the most valuable skill for using these tools in the workplace. #DataEngineering #ModernDataStack #CareerDevelopment #TechSkills #DataInfrastructure

I just recorded a "how to deploy dlt to dagster demo" as part of our upcoming course. Since Dagster is easy peasy, the whole thing takes only 10 minutes. My microphone though decided to be a diva and forced me to re-record a handful of times. Want to get notified when the course comes out? Sign up to our education newsletter: https://lnkd.in/eRfYu-Gs

Airlift is a powerful new Tookit that makes it easy to bring all of your Airflow DAGs into the Dagster control plane. Without modifying your Airflow DAGs, you get the operational and observability benefits of Dagster, including: ?? Rich Metadata and run history ?? Data Quality checks ?? Dependency mapping and automation Instead of a lengthy migration project you get immediate value to build momentum towards building a more powerful Data platform. Check out Airlift today!

Great collection of links from Tim Hopper about how other developers are using LLMs to augment their work, including interviews with Jowanza Joseph, Ravi Mody and Simon Willison https://lnkd.in/gdBamkbS