What is Azure Data Explorer?

Azure Data Explorer (ADX) is a fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites and IoT devices.

The primary use of ADX is the ingestion of structured, semi-structured and unstructured data for big data analytics, with speeds of up to 200 Megabytes/sec per node (up to 1000 nodes) returning results in less than a second across billions of records.

More businesses are opening their network to a wide variety of IoT devices and applications, it becomes increasingly vital for businesses to proactively react to events in a timely and cost-effective manner.

I recently employed ADX with a government client to migrate an existing Kafka workload which ingests and transforms Fortinet, Paloalto, and Bluecoat web security logs. During Covid-19, their workload increased 10-fold, with an associated 5-fold increase in costs. The migration of this workload resulted in a 60% cost reduction, a simplified solution and an improvement in data reliability.

How can data be ingested into Azure Data Explorer?

Azure Data Explorer supports server-side stored functions, continuous ingest, and continuous export to Azure Data Lake store. It also supports ingestion time-mapping transformations on the server side, update policies, and precomputed scheduled aggregates with materialized views.

Automated Pipelines - Ingestion Methods

• Event Grid Blob Created - When a 'blob' is created on the Azure storage account it results in the firing of an event that triggers the Data Explorer ingestion pipeline.
• Event Hub
• IoT Hub
• Azure Data Factory
• Light Ingest - Command line tool for historical loads to minimise cost.

Supported Formats

• Uncompressed Formats - ApacheAvro, AvroCSV, JSON, MultiJSON, ORC, Parquet, PSV, RAW, SCsv, SOHsv, TSV, TSVE, TXT, W3CLOGFILE?????

When the source data has a schema provided e.g. avro, parquet, w3clogfile it can be directly inserted into the final destination table with the expected data types, column names etc.

• Compressed Formats - GZip, Zip

Transformations

Data is transformed in ADX by using the native language KQL - Kusto Query Language. This is a simple, yet powerful language to query structured, semi-structured and unstructured data. It assumes a relational data model of tables and columns, with a minimal set of data types. The language is very expressive, easy to read and understand the query intent.

Visualisations

Use different visual displays of your data in the native Azure Data Explorer?Dashboards. You can also display your results using connectors to some of the?leading visualisation services, such as?Power BI?and?Grafana. Azure Data Explorer also has?ODBC?and JDBC connector support to tools such as?Tableau?and?Sisense.

Use Cases

For Fortinet web security log files using ADX click here.

Final Thoughts

I hope you have found this helpful and will save your company understand the basics of Azure Data Explorer.

Please share your thoughts, questions, corrections and suggestions, please drop me a message on?LinkedIn.