GRiT FOOTPRiNT #3 - Data Warehouse

In this month’s GRiT FOOTPRiNT, we dive into a project in which the main objective is to create a data warehouse where files can be uploaded and deployed to go to other dashboards, machine learning, platforms, and others.

Our client is deploying data from an SQL Server into ADSL as server. GRiT is implementing automation and refactoring part of a platform to adapt it into the new data flow.?

Here are the highlights of this project. ??

Project Scope

The project’s main objective is to create a data warehouse where files can be uploaded and deployed to go to other dashboards, machine learning, platforms, and others. Moreover, the team is responsible to refactor a fraction of the platform, in order to be adapted to the new data flow.

?

Methodologies and Technologies

GRiT currently has two consultants working in this project, a Machine Learning Engineer, that doubles as Data Engineer, responsible to automate processes, and a DevOps professional, supporting the project and the team as a DataOps Engineer. These consultants operate within the client’s internal team, which is a very heterogeneous one, allowing professionals to work closely and collaborate more effectively. This is especially relevant because they work based on Agile methodology.

The technologies used to develop this data warehouse are Azure Data Factory and Databricks.

?

Project Flow and Results

The uploaded data are then forwarded to analysis services, Power BI professionals, and analysis teams to analyse them.

Before hand, everything was on-premise, in SQL Server. The idea was to move this process to the cloud, using ADSL as storage. Furthermore, to effectively do this transfer, all files and data are processed using Databricks clusters.

For the client, this process will allow them to reduce costs, and ultimately automate most data processes.