Monitoring and Analyzing Pipeline and operations in Azure Data Factory
Azure Data Factory, a tool in Azure helps Orchestrate and control the data between different systems. Data Factory consists of Pipelines and other sets of elements associated with it like Integration Runtime, etc. Once the Pipeline is designed, it is important to monitor its execution.
There are multiple ways to monitor the operations and Data Factory V2 comes with additional information like Integration runtimes, Triggers, etc., and are simplified into simple User Interface.
Monitoring Dashboard
The Monitoring Dashboard can be accessed through the Data Factory Console. The Author and Monitor tile in the Data Factory Blade open the Data Factory console.
The Data Factory Blade also shows the different count of execution and status along with Integration runtime resource consumption.
In the Data Factory console, the monitoring tab can be selected using the . The monitoring Dashboard shows different views for Pipeline runs, Integration runtime and Trigger runs.
Pipeline Executions
The Pipelines execution can be filtered based on Time and Time zones. Each Pipeline executions shows additional details such as duration, status, parameters, and the error message if it had been failed.
The execution can be re-run and it also provides information about the activity for each execution.
Integration Runtimes
The Integration Runtimes shows information about the Integration Runtimes that are configured and also the status & type for the same.
The Pipeline activities for each of the Integration Runtimes can also be monitored.
Trigger Runs
The Trigger runs show the Pipeline execution information for each Trigger. It shows information about the Trigger such as Tigger Name, Status, Trigger type and the Pipeline which is associated with it. The View is like Pipeline execution.
Alerts
The alerts help notify based on various signals for Azure Data Factory. Alerts for Data Factory are configured as part of the Azure Monitor Alerts.
Alerts can be used to notify on the action or act upon it. Various actions such as Azure functions, Logic Apps, Web hooks, etc. and can be used to automate the post-execution action such as performing operations on the Data.
Metrics
Different metrics about the Data Factory Pipelines can be visualized from the Monitoring Metrics in Azure. Multiple metrics can be compared to build a customized view and it can be exported into Excel for advanced analysis. Various metrics such as Pipeline run metrics, Integration Runtime usage, etc. can also be measured.
Monitoring is an important part of the Azure Data Factory Pipeline. Proper alerts and metrics can help in the smooth operation of the Pipelines.