Triggers in ADF
Kumar Preeti Lata
Microsoft Certified: Senior Data Analyst/ Senior Data Engineer | Prompt Engineer | Gen AI | SQL, Python, R, PowerBI, Tableau, ETL| DataBricks, ADF, Azure Synapse Analytics | PGP Cloud Computing | MSc Data Science
Azure Data Factory (ADF) is a powerful cloud-based data integration service that allows you to create, schedule, and orchestrate your data workflows. One of the key features that makes ADF so versatile is its ability to automate pipeline executions using triggers. In this article, we'll explore the different types of triggers available in ADF and how to use them to streamline your data operations.
Types of Triggers in ADF
1. Schedule Triggers
Schedule triggers are designed for scenarios where you need to run a pipeline at specific times or on a regular interval. They are ideal for routine data processing tasks, such as daily or hourly data refreshes.
Configuration Options:
Example: To create a schedule trigger that runs every day at 6 AM:
{
"name": "DailyTrigger",
"properties": {
"type": "ScheduleTrigger",
"typeProperties": {
"recurrence": {
"frequency": "Day",
"interval": 1,
"startTime": "2024-07-15T06:00:00Z",
"timeZone": "UTC"
}
}
}
}
2. Tumbling Window Triggers
Tumbling window triggers are similar to schedule triggers but are designed for processing data in contiguous, non-overlapping time intervals. They are perfect for batch processing scenarios where you need to process data in fixed time chunks, such as hourly or daily windows.
Configuration Options:
Example: To create a tumbling window trigger that runs every hour:
{
"name": "HourlyTrigger",
"properties": {
"type": "TumblingWindowTrigger",
"typeProperties": {
"frequency": "Hour",
"interval": 1,
"startTime": "2024-07-15T00:00:00Z",
"timeZone": "UTC"
}
}
}
领英推荐
3. Event Triggers
Event triggers run pipelines in response to specific events, such as when a blob is created or deleted in Azure Blob Storage. This type of trigger is useful for real-time or near-real-time data processing scenarios.
Supported Events:
Configuration Options:
Example: To create an event trigger that runs when a new blob is created:
{
"name": "BlobCreatedTrigger",
"properties": {
"type": "BlobEventTrigger",
"typeProperties": {
"scope": "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>",
"events": ["Microsoft.Storage.BlobCreated"],
"blobPathBeginsWith": "/<container>/<path-prefix>/",
"blobPathEndsWith": ".csv"
}
}
}
4. Manual Triggers
Manual triggers allow you to run a pipeline on-demand. This is useful for ad-hoc scenarios where you need to manually start a pipeline execution.
Usage:
Example: To manually trigger a pipeline execution from the ADF UI:
Conclusion
Triggers in Azure Data Factory provide a flexible and powerful way to automate your data workflows. Whether you need to run pipelines on a schedule, process data in fixed windows, respond to real-time events, or manually kick off executions, ADF has a trigger type that fits your needs. By leveraging these triggers, you can streamline your data operations and ensure your data processes run smoothly and efficiently.