Mage

We're saddling up for a wild ride with our 1923 Release! This release is all about taming the data frontier and building a robust foundation for your data-driven future! ???? ? Here's what's new: ?? Enhanced API source for more robust data integration ?? Teradata Source and destination support ?? Hybrid Deployment on Microsoft Azure for flexible resource management ?? Apache Iceberg destination for modern data lakehouse architectures ? Airbyte Cloud Integration for seamless data ingestion ?? Comprehensive Databricks Integration Suite ?? Universal Block Concurrency Limit for optimized performance ???AI Inline code assistance ?? AI data-aware autocomplete Full release notes ?? https://lnkd.in/g8jZEeqv

?? Join us for our next Lunch & Learn next Thursday, March 20, at 12:00 PM PST. Mage Co-founder & CEO, Tommy Dang, will explore how Mage Pro's multi-tenant architecture powers Data Mesh excellence. Learn how to enhance scalability, security, and efficiency in managing diverse data needs across multiple tenants. You won't want to miss out! ?? #datamesh #dataengineering #mageai #lunchandlearn

Take your coding to the next level with AI! ?? ?? Say goodbye to endless debugging and searching for solutions. With?Mage Pro's AI Inline Code Assistance, coding just got smarter. Designed for data engineers and ML practitioners, this intelligent assistant provides: ??Real-time, context-aware suggestions ??Error detection and automated fixes ??Optimized transformations and instant documentation Whether you're crafting SQL queries, transforming data, or debugging pipelines, Mage Pro streamlines your workflow and ensures best practices—all while reducing costly errors. Learn more ???https://lnkd.in/dQGrp9vf #DataEngineering #AI #MachineLearning #CodingEfficiency #MageAI

?? MageAI is revolutionizing my data pipeline workflow! ?? As an open-source solution for data piping, MageAI brings the best of notebooks with the power to run SQL, Python, and R seamlessly. Setting up a pipeline is as simple as writing a YAML file, and the best part? You can connect your favorite AI to generate the pipeline for you! If you're looking for an intuitive, flexible, and AI-powered way to streamline your ETL and data workflows, this tool is a game-changer. ?? Have you tried MageAI yet? Would love to hear your thoughts! ?? #DataEngineering #AI #ETL #MageAI #MachineLearning

To make data-driven decisions we need to collect siloed data, combine and make it available for analysis. This requires multiple tasks to be run in a particular order. Here workflow orchestration tools help us. These tools provide real-time visibility into the status and progress of workflows, enabling organizations to track performance, identify bottlenecks, and make data-driven decisions for process improvement. In my latest blog, I list popular orchestration tools in data engineering. Link in the comment. #opensource #dataengineering #airflow #mageai #dagster #temporal

???My Experience with Mage AI: Simplifying Data Pipelines! I recently started using?Mage AI?for building data pipelines, and I have to say — it’s been a game-changer! As someone who’s dealt with the headaches of complex ETL processes, I was blown away by how intuitive and efficient Mage AI’s?Blocks?are. Here’s what stood out to me: ??Modular Workflows: Blocks are like Lego pieces — you can snap them together to create powerful pipelines. No more reinventing the wheel! ??Flexibility: Whether I’m loading data from PostgreSQL, transforming it with Python, or exporting it to Snowflake, Mage AI makes it seamless. ??Debugging Made Easy: Each Block operates independently, so troubleshooting is a breeze. ??dbt Integration: As a fan of dbt, I loved how easily I could incorporate dbt models into my workflows. The drag-and-drop interface is a cherry on top — it’s so satisfying to see everything come together visually. I’ve shared more details about how blocks in Mage AI works in my?Medium article?— check it out here: https://lnkd.in/dB7mVArp Have you tried Mage AI or any other tools that simplified your data engineering process? Let’s share experiences in the comments! ?? #DataEngineering #ETL #DataPipelines ##MageAI #DataTransformation #WorkflowTools

Still wanting to add an interesting project to your Data Analytics portfolio? Check out what’s been completed on my end-to-end golf analytics project. Over the weekend I was able to fetch player level data from the DataGolf API and then sync it to a BigQuery table using Mage sql blocks. Here’s what I completed so far: ? Fetched hole level player shot data ? Synced my pipeline to BigQuery ? Dumped raw data to Bronze layer Over the next few days, I’ll probably change this to an incrementally loading query so that I can reduce the amount of space I’ll need for storage in BigQuery. I’ll need to complete this incremental load to eliminate duplicates in the Silver layer anyways. I’ll also add some cool SQL to the Silver layer, think LAG and some advanced window functions. My goal is to have a MVP dashboard for you by Sunday of the Players Championship. Thats this week! What’s your view on this project, good for a data engineering / analytics portfolio? Let me know what you think in the comments. If you're finding value in my content, please consider reposting ?? and following me for more insights on SQL, Mage, and data analytics. #dataengineering #dataanalytics #sql

Struggling to integrate SFTP data into BigQuery? ?? Need SFTP data for your data warehouse? ?? Moving data into BigQuery can be complex. Mage Pro provides a streamlined solution and automates data pipeline creation. ? Simplify your workflow and improve data availability. Explore the guide in this article by our DevRel, Cole Freeman! ???? #dataengineering #dataintegration #bigquery #mageai

Yesterday I finished up recording a tutorial showing how to connect Mage Pro SQL blocks with Databricks. Thought I would share it with y’all. In the video, I walk through the entire process: ?? Creating a new pipeline ?? Pulling golf rankings data from an API ?? Setting up your Databricks configuration in Mage Pro ?? Writing the SQL code that creates tables in Databricks I’m using a similar setup for my golf analytics project using BigQuery, but I wanted to showcase Mage’s new integration with Databricks. What's the best part? Once you get this working (takes about 10 minutes), you can automate data flows from virtually any data source into your Databricks tables. Check out the full video to see how I did it step-by-step. If you're building data pipelines and want to simplify your workflow, this one's worth a watch! What data integration challenges are you tackling? Let me know in the comments! #dataengineering #dataanalytics #databricks #sql

So you pulled some data from an API and think you're ready to build dashboards? Not so fast! Let's talk about the bronze layer. If you are following along we fetched some data from the GolfData API earlier this week. It’s now time to store our raw data in Google bigquery. Think of the bronze layer as your data's landing pad, it's still messy and definitely not dashboard-ready. For my fellow data nerds, here's what's happening in my bronze layer setup: ? Dumping raw API responses with minimal modifications ? Adding timestamps to track when each record arrived ? Preserving all fields, even the ones I'm not sure I'll need ? Creating unique identifiers for easier tracking downstream So what’s next? Transforming this messy data into something useful in the silver layer. That's where the real magic happens. What's your approach to data layering? Bronze/silver/gold or something totally different? Drop your thoughts below! If you're finding value in my content, please consider reposting ?? and following me for more insights on SQL, Mage, and data analytics. #dataengineering #dataanalytics #sql #golf