Azure Event Hubs is a distributed stream processing platform and event ingestion service managed by Microsoft. Event Hubs gives organizations a fully-managed solution to receive and process millions of events per second (without concerning about infrastructure and SLA). Azure Event Hubs is truly distinctive vs. Apache Kafka as it has a seamless integration with other services in Azure.
Event hubs is a highly scalable telemetry service offering one-way communication with the HTTP/AMQP protocol. You can send events from anywhere: a website, an app, an IoT device, a software, etc. Azure Event Hubs is distinct from Azure IoT Hub as communication is one way and not two ways.
Event Hubs is the component to use for real-time and/or streaming data use cases:
- Real-time reporting
- Capture streaming data into files for further processing and analysis – e.g. capturing data from micro-service applications or a mobile app
- Make data available to stream-processing and analytics services – e.g. when scoring an AI algorithm
- Telemetry streaming & processing
- Application logging
- An Event Hub Producer Client is a source of telemetry data, diagnostics information, usage logs, or other log data, as part of an embedded device solution, a mobile device application, a game title running on a console or other device, some client or server-based business solution, or a web site.
- An EventHub Consumer Client picks up such information from the Event Hub and processes it. Processing may involve aggregation, complex computation, and filtering. Processing may also involve distribution or storage of the information in a raw or transformed fashion. Event Hub consumers are often robust and high-scale platform infrastructure parts with built-in analytics capabilities, like Azure Stream Analytics, Apache Spark, or Apache Storm.
- A partition is an ordered sequence of events that is held in an Event Hub. Azure Event Hubs provides message streaming through a partitioned consumer pattern in which each consumer only reads a specific subset, or partition, of the message stream. As newer events arrive, they are added to the end of this sequence. The number of partitions is specified at the time an Event Hub is created and cannot be changed.
- A consumer group is a view of an entire Event Hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and from their own position. There can be at most 5 concurrent readers on a partition per consumer group; however, it is recommended that there is only one active consumer for a given partition and consumer group pairing. Each active reader receives all of the events from its partition; if there are multiple readers on the same partition, then they will receive duplicate events.
The W’s section emphasizes on three pillar actions “Why, When and Where” Azure Event Hubs comes into play.
- Azure Event Hubs allows you to raise a data pipeline capable of processing a huge number of events per second with low latency.
- It can process data from parallel sources and connect them to different infrastructures and services.
- It supports repeated replay of stored data.
- To validate many publishers and to save the events in a Blob Storage or Data Lake.
- When you want to get timely insights on the business application.
- To obtain reliable messaging or flexibility for Big Data Applications.
- For seamless integration with data and analytics services to create big data pipeline.
- Anomaly Detection
- Application Logging
- Archiving data
- Telemetry processing
- Live Dashboarding
