How does LinkedIn use Kafka to process events?
Kafka Application

How does LinkedIn use Kafka to process events?

Akash Sur Akash Sur

Akash Sur

AVP Data Management at Commonwealth Bank of Australia | CSM? | DEI Advocate

发布日期: 2024年5月15日
+ 关注

At LinkedIn, Kafka is used for various purposes such as logging, data replication, data processing, and data warehousing. Each data center where user-facing services run has a Kafka cluster that receives log data from frontend services in batches, which is then consumed by online consumers within the same data center.?

A?separate Kafka cluster is deployed in a geographically close data center for offline analysis, where data load jobs pull data from the replica cluster of Kafka into Hadoop and data warehouse for reporting and analytical processes. Kafka is also used for prototyping and ad hoc querying.

要查看或添加评论，请登录

Akash Sur的更多文章

  • HDFS Simplified: Architecture and important terminologies
    2024年5月29日

    HDFS Simplified: Architecture and important terminologies

    The HDFS architecture and different types of operations have been shown in the figure attached. Below are some…

  • HDFS Simplified: Blocks and Nodes
    2024年5月28日

    HDFS Simplified: Blocks and Nodes

    HDFS stands for Hadoop Distributed File System. It is the primary application to handle and store bigdata in Hadoop.

  • Hadoop Ecosystem Applications
    2024年5月24日

    Hadoop Ecosystem Applications

    In this article, we will discuss some commonly used applications of Hadoop Ecosystem in brief based on the following…

  • HBase Simplified: Architecture
    2024年5月23日

    HBase Simplified: Architecture

    Main Components: HMaster Region Servers Region Zookeeper HMaster: Monitors the Region Server instances Assigns regions…

  • HBase Simplified: Features
    2024年5月22日

    HBase Simplified: Features

    HBase is a column oriented non-relational database management system. It runs on top of HDFS.

  • Hive Simplified: Clients and Services
    2024年5月21日

    Hive Simplified: Clients and Services

    Hive Clients: JDBC Client - allows Java applications to connect to Hive ODBC client - allows applications based on ODBC…

  • Hive Simplified
    2024年5月16日

    Hive Simplified

    What is Hive? -- Data Warehouse Software within Hadoop Ecosystem -- Designed for reading, writing and managing tabular…

  • Kafka Simplified: How does Kafka Store logs ?
    2024年5月14日

    Kafka Simplified: How does Kafka Store logs ?

    This example topic has four partitions P1–P4. Two different producer clients are publishing, independently from each…

  • Kafka Simplified: Introduction to Apache Kafka
    2024年5月13日

    Kafka Simplified: Introduction to Apache Kafka

    Kafka uses a publish-subscribe (push to store logs and pull to consume) messaging model, where data is organized into…

社区洞察

其他会员也浏览了