Astronomer

Attending Amazon Web Services (AWS) re:Invent next week? Stop by booth #466 to join our #GenAI DAG Scavenger Hunt, learn about the ongoing public preview of Astro Observe and get hyped for the upcoming release of Airflow 3.0. ? Simplify your Apache Airflow management, increase operational efficiency, ensure reliable data delivery and more with Astro. ?? Meet with us: https://bit.ly/40Kgt1a #AWSreInvent

As Pete DeJoy shares with SiliconANGLE & theCUBE at the NYSE, AI outcomes aren't constrained by how fast you can prototype. Truly differentiated AI results are determined by two key questions: ?? Are you solving high-context problems that are strategic to your business outcome? ?? Do you have access to domain-specific and trustworthy data? For example, Ford Motor Company is using Astro and Apache Airflow to train their Advanced Driver Assistance Systems (ADAS), orchestrating diverse, compute-intensive pipelines and processing terabyte-scale daily data from lidar, radar and video. Click here to learn more about Ford's use of Astro and Airflow for ADAS: https://bit.ly/3CPEdHi

?? On Tuesday night, we took a moment to celebrate our 15 year anniversary with our portfolio founders and members of the media who've helped write the story of Salesforce Ventures through the years. Thank you to all who attended: Astronomer, Altana, Sixfold, Writer, Runway, ElevenLabs, Axios, CNBC, Fortune, Forbes, Reuters, TechCrunch, The Information, VentureBeat, and The Wall Street Journal. We look forward to writing our next chapter alongside all of you. For more on the past, present, and future of Salesforce Ventures,?read Paul Drews' reflections on our blog: https://lnkd.in/ggt2yaAm

Last year, Stripe processed over $1 trillion, equivalent to 1% of global GDP. ?? To balance strict regulatory compliance with keeping its developers nimble, Stripe operates Apache Airflow at massive scale: ?? Airflow processes multiple petabytes of data every day ?? Airflow orchestrates 250 complex pipelines comprising 150,000 tasks that connect Stripe’s operational and analytical systems ?? Airflow manages workflows across Apache Spark with data transformed into the Apache Iceberg table format and queried via Presto for 500 different teams Learn more about Airflow's criticality within Stripe: https://bit.ly/4eMXk1X

Is your data pipeline management as seamless as it could be? Serjesh Sharma of Ford Motor Company uncovers how they leverage both open-source and managed versions of Airflow, like Astronomer, to gain a clear, unified view of their data operations. He explores how this approach helps Ford efficiently manage and support their entire data pipeline, ensuring everything runs smoothly from a single interface. Click the link in the comments for the full episode. #AI #Automation #Airflow #MachineLearning

Are you staying ahead in data management innovation? In our newest podcast episode, Ben Tallman from M Science dives into the emerging tools in Airflow that are making it easier for businesses to manage and query data across their entire systems. Benjamin explains how these advancements can simplify complex processes and improve overall efficiency. Listen in to learn more about these innovations and how they can enhance your data operations. The link is in the comments. #AI #Automation #Airflow #MachineLearning

What a week it's been here at Microsoft Ignite! From Astronomer ice cream ?? to space-themed the LEGO Group raffles (plus plenty of Astro demos, of course), it has been all systems go at our booth #422. ?? If you haven't already, make sure to stop by before heading out to learn how Astro - an #Azure Native ISV Service powered by Apache Airflow - can help you manage and scale your data operations with Microsoft Azure. https://bit.ly/412x0xX #MicrosoftIgnite2024

"We started with 25 teams and now have over 50 teams using Airflow. Astronomer has really helped us scale. The fact that we're not seeing the failures we experienced with Oozie is a testament to how much smoother things are running now." - Bhavesh Jaisinghani, Data Engineering Manager at Autodesk. To keep pace with rapid growth and the increasing complexity of Autodesk’s data workflows, Autodesk's data licensing team needed a more efficient, scalable, and user-friendly orchestration solution. The company turned to Astro, powered by Apache Airflow, to transform their data engineering processes. Learn how Autodesk leveraged Astronomer and Apache Airflow to streamline and scale their data workflows: https://bit.ly/3UZB8uu

Tomorrow! ?? If you're at Microsoft Ignite, swing by our happy hour at Lazy Bird at the Hoxton. RSVP today: https://bit.ly/3C6CZHl #MicrosoftIgnite2024

Apache Airflow in Action: Scaling Insights from Bosch and 1.2 Million Pipeline Runs Per Day - with 1 second latency In their initial Airflow environment, Bosch used an event-driven pattern. Data was ingested from Bosch’s internal file repository to the Microsoft Azure Event Grid pubsub service, then into Airflow. However, a series of event storms meant tasks queued, impacting freshness of data outputs. As these event storms became more frequent, the team at Bosch had to prioritize Airflow scale. Learn how Bosch tuned its Airflow setup to go from 1,000 to 50,000 DAG runs per hour, along with best practices and anti-patterns observed along the way. https://bit.ly/3Zgpcr3 #DataEngineering #DataOps