Mage的动态

Transcript

Are you looking to enhance your digital workflows by connecting your data pipelines directly to Databricks? Match Pro SQL Blocks may be the solution you've been waiting for. Hi, I'm Cole, a Developer Advocate at Mage, and today I'm excited to walk you through one of our newly released features, Match Pro SQL Block Integration with Databricks. In this video, I'll demonstrate how many Pro makes it simple to connect your data pipelines to Databricks using our SQL blocks. We'll cover everything from setting up the connection to creating automated pipelines that keep your data. Like environment In Sync with your source data systems. By the end of this tutorial, you'll have a powerful, reliable system for leveraging the full potential of your data from your source environment to Databricks. Let's get started. So here we are in our macro environment and the first thing we want to do is set up our Databricks configuration. So we will hover over this left hand navigation menu and head over to our editor page. We'll wait for our files to populate, and then we'll Scroll down until we see our aisle config dot yaml file. We'll click this open, and you can see that we have an integration here with Databricks. We'll want to add our Databricks access token, our database that we're working in the host. The Databricks HTTP path and the Databricks schema. Now most of these are set up in secrets so that you or nobody else can have access to my system, but you'll want to do that the same for yours. So once we get our configuration setup complete, we can navigate over to our pipelines page where we can go ahead and create this data pipeline. So we'll head back over to our left hand navigation menu, hover over and we'll click the Pipelines button. We'll go ahead and create a new pipeline here and we'll start from scratch. We're going to create a batch pipeline, so we'll go ahead and give it a name. We'll call this data Bricks. Video. The description is gonna be simple databricks. Pipeline using SQL blocks and we can tag it whatever we want, tag it by name and demo and that's it. So we'll go ahead and click this Create New Pipeline button. And then we will come into our pipelines page. If you want, you can go ahead and expand these sidebars out so you can see your files that you're working with. Once you also create some blocks, you can see the dependency tree. So we will go ahead and create our first block. So it's going to be a data logger. We're gonna select the API, We're going to call this Fetch World Golf Rankings. API data so we'll save an AG and I already have some code prepared so I can go ahead and copy this. Let's copy this code here and the data loader block so you can see that we are needing access to an API key and we're using this space URL here and we are going to return a pandas data frame. So you can see that this is a secret. If you want to add secrets, you can navigate to the right hand pop out menu and hit add. The secret and you can provide it a key value pair, hit enter and you'll have a new secret. You can see that I am storing this data Goff API key right here. So let's go ahead and hit the tree here so we can get a good view of this dependency tree and then we're going to go ahead and run this block. And you can see that it returns a sample of your data. There's, there's 10 rows here. And this is just, this is just World Golf ranking data. So it's got the player name, the ID from the API that it comes from, and some other descriptive data about what 2 are they play for where they're from. And the time it was loaded, so we can go ahead and reduce that output there. And we will add our second block here, which is a SQL block. Again, I have some code I'm going to copy in and we will go ahead and connect to Databricks. So let's go ahead and select the transformer. We're going to select our base template for our SQL block. We'll hit enter and we'll call this. We'll call this Databricks SQL connection. And then we're going to hit save and add. Let's go ahead and copy in the code. So I'm going to create or replace the table, this World Golf rankings table. Here we're just selecting our basically all of our columns from the previous return. And you can see that here we're selecting from DF one. And if you look at your, if you look at some of your retirement here and the header, you'll see that DF one is the fetch. World Golf Rankings API data block, and that's this block above here. That's your upstream block. So what we're going to want to do next is go ahead and provide a connection. So we are connecting to Databricks, we'll select Databricks, we'll give it a profile. I did have our our YAML configuration configured to this Databricks profile. So we'll select Databricks and we're going to use raw SQL here. So basically what we're doing is we are selecting everything in this CTE from the upstream block. And then we are just doing a simple case when statement here. And then we'll select, we'll select everything from that convert amateur block or convert amateur CTE. So let's go ahead and run the code. Once we run this code, we should see that our. Code not only runs here in Manage, but also is integrated with our Databricks resource. So we'll give this a second to Ron. So now that we see that our block has completed running, we can navigate over to our data bricks, our Databricks instance, and we can see that our World Golf rankings formatted. Table is here in in databricks. And we can actually query this. We can actually query this data here. So let's just go ahead and do account star statement. So we're going to run select. Count Star. From workspace. The default. Dot world. Golf rankings formatted. And we'll run this. And you can see that we returned 500 rows. We can also just select from the top 10 so we can do a select. Or select star from workspace. Let's just copy this part up here. And we'll just limit this to 10. Then run this. You can see that our Dega has run here in Databricks and it looks like we have a successful integration here. That's it from me today. I hope you enjoyed this video and I hope this helps you out whenever you're trying to integrate your Mage Pro pipelines with data bricks. Thanks and have a great day.