Ingesting Data From REST API endpoints in Fabric Lakehouse
Technical Details
Representational?State?Transfer (REST) is a software architectural design that has been employed throughout the software industry to create?stateless web applications. An application that adheres to the?REST may be informally described as?RESTful.? This term is more commonly associated with the design of?HTTP-based?APIs?to interact with data that is stored on a vendors web site.? Today, we are going to explore how Python can be used in a Fabric notebook to interact with a popular website called Spotify.
Business Problem
Our manager has asked us to create a prototype notebook that can be used to interact with a web site using the REST API endpoints.? This code will demonstrate the following concepts:? how to grab an access token; how to perform a single read from an endpoint; how to save simple JSON data to a file; how to perform paged reads from an endpoint; and how to convert multiple JSON files into a single delta table.? To make this code somewhat reusable, we will want to write python functions.
领英推荐
Spotify Data
To access most REST API services, we need to login to the service to retrieve a bearer (access) token.? This token usually has a lifespan of 60 minutes.? During that time, multiple calls to various endpoints can be made.? Once the token expires, an HTTP error will occur.? To resolve this error, just ask for another token.
I like using a modular design when coding in Python.? You can even take it one step further by using object-oriented programming.? Each task is broken into functions.? I created functions for the following tasks.
Each one of these tasks were coded into a function and tested separately.? These components were called one or more times to load both the genres and artists entities.? Today’s article was not meant to be a complete coverage of the Spotify REST API.? Instead, its purpose was to show design patterns that you might encounter when writing Python code in Microsoft Fabric to talk to Web Services.