Microsoft Fabric: Centralized Python Utilities

Are you looking for a streamlined way to share and distribute Python utilities across different environments in Microsoft Fabric? In this post, I’ll walk you through a proposed solution for deploying a Python wheel to Microsoft Fabric. This method aims to provide a centralized repository for your utilities, ensuring consistency and efficiency across development, staging, and production environments.

You can find the code on GitHub: fabric-python-utilities

This template offers a programmatic, Infrastructure as Code (IaC) approach to deploying a wheel to Microsoft Fabric. Please note, this post and the associated code deploy to one environment; feel free to adjust it for your multi-environment setup.

Note: In the future, consider uploading the library to a Microsoft Fabric environment using the REST API. But as I understand it, the API does not support Service Principals yet. I believe that this approach soon (hopefully) will be outdated, but this can maybe help you until Microsoft supports more out-of-the-box approach.

Benefits

Centralized Utilities Repository

• Maintains consistency across projects.?
• Reduces duplication of code.?
• Simplifies updates and maintenance.
• ?Enhances discoverability of utilities for developers.?

Sharing Utility Functions

• Promotes collaboration across teams.?
• Ensures best practices are shared and implemented.?
• Saves time by reducing the need to re-develop common functionalities.?
• Standardizes approaches to common tasks.?

Distributing a Python Wheel

• Simplifies distribution and installation with pip.?
• Facilitates version control and dependency management.
• Streamlines updates to utility functions.?
• Enhances manageability of utility functions across workspaces.

Prerequisites and notes

First, you need to assign the deploying service principal as “contributor” to the workspace. Navigate to “Manage access” and add the SPN. It is recommended to automate this process using IaC. More details can be found here: Give Access to Workspaces.

With contributor access, you can upload the wheel using AzCopy to copy the wheel into OneLake. Refer to the AzCopy documentation for more information: Using AzCopy.

Ensure the utilities lakehouse is set as the default. Installing from a non-default location may not work.

The flow

When using pip install inside a notebook, follow the Fabric Python inline installation guide: Python Inline Installation.

Add the _inlineInstallationEnabled from the pipeline and remember to toggle parameter inside the notebook.

Notebook example

I will give you an overall idea of how to apply the utilities inside a Microsoft Fabric notebook.

First, ensure that the inline installation is enabled (from the pipeline):

_inlineInstallationEnabled=True        

Installing using pip:

%pip install --force-reinstall /lakehouse/default/Files/PythonUtilities/fabricutilities-1.0-py3-none-any.whl        

Restart python (might not neccesarry):

mssparkutils.session.restartPython()        

You can verify the pip installation:

%pip show fabricutilities        

Finally, you can use the python library:

from company.utilites import dummy_function        

Install from non-default lakehouse

I have found a way to install the wheel from a non-default lakehouse using the notebook /built-in/ path:

import shutil 
import os 
from pyspark import SparkFiles 

_wheel_path = "abfss://<workspace-name>@onelake.dfs.fabric.microsoft.com/<lakehousename>.Lakehouse/Files/PythonUtilities/fabricutilities-1.0-py3-none-any.whl" 

sc.addFile(_wheel_path) 

_src_path = SparkFiles.get("fabricutilities-1.0-py3-none-any.whl") 

_target_file_path = mssparkutils.nbResPath + "/builtin/fabricutilities-1.0-py3-none-any.whl" 

os.remove(_target_file_path) 

shutil.move(_src_path, _target_file_path)        

After this, you can %pip install:

%pip install builtin/fabricutilities-1.0-py3-none-any.whl        

Challenges

• Be aware, that you need to add the utilities lakehouse as the default. I have tried installing from a non-default, but have not managed to get it to work.
• When using the pip install inside a notebook, remember to enable inline installation
• Probably at lot more challenges....

Links

• Use AzCopy with Microsoft Fabric: https://medium.com/microsoftazure/ingest-data-into-microsoft-onelake-using-azcopy-a6e0e199feee
• Inline installation: https://learn.microsoft.com/en-us/fabric/data-engineering/library-management#python-inline-installation
• AzCopy: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10
• Workspace access: https://learn.microsoft.com/en-us/fabric/get-started/give-access-workspaces
• Microsoft Fabric and wheel deployment: https://learn.microsoft.com/en-us/fabric/data-engineering/library-management