How does the publish-subscribe architecture work in event-driven systems?

The Pub - Sub aka Publisher and subscriber model is a one of the messaging/communication pattern which is most common in Event driven distrubted microservice architecture. A publisher is one publishes a message to the broker. Broker acts as a middleman, subscriber subscribes to specific events against broker. If a publisher posts an event to message broker it gets the list of subcribers for that event and make sure to propgate those events to all the subscibers. Then subscriber can do thier own logic if it has been invoked. Broker is responsible for managining topics, evenrs and also delivery of events to subscriber. This pattern allows loose coupling between receiving and processing therby increasing throughput, scalablity and reliablity.

The Publish-Subscribe (Pub-Sub) messaging pattern has significantly impacted event-driven system architecture in my experience. It facilitates communication between independent components by enabling publishers to send messages (events) to a central broker, which then distributes them to relevant subscribers based on their specified interests. This decoupling approach provides numerous benefits, including: Loose coupling: Publishers and subscribers operate independently, oblivious to each other's details. Scalability: The central broker efficiently manages message delivery, allowing for easily scalable systems. Fault tolerance: The system remains operational even if individual components fail due to the distributed nature of event delivery.

The publish-subscribe (pub-sub) architecture in event-driven systems is a messaging pattern where senders of messages, called publishers, do not specifically program the messages to be sent directly to receivers, called subscribers. Instead, the programming model relies on the concept of topics or channels that messages are published to. Subscribers express interest in one or more topics, and only receive messages that are of interest to them. This decoupling between publishers and subscribers allows for greater flexibility and scalability in systems.

PubSub system design pattern allows you to improve scalability of your system and especially go through high load spikes without pain. You should consider it when designing your next big thing

Pub/Sub stands for "Publish/Subscribe" and refers to a messaging pattern wear senders of massages (publishers) do not program the messages to be sent directly to specific receivers (subscribers). Instead, the messages are published to topics, and subscribers express interest in one or more topics, receiving only the massages relevant to them. It's a way of decoupling massage senders from massage receivers, allowing for more flexible and scalable communication systems.

In designing a pub-sub system within a .NET Core microservices architecture, your primary focus should be on selecting a suitable messaging platform that integrates well with .NET Core, such as Azure Service Bus, RabbitMQ, or Kafka. Each offers distinct advantages, so your choice should align with your system's scalability, reliability, and performance requirements

There are many considerations of pub-sub: - Does the ordering need to be maintained - What is the failover mechanism? - Rate of publish and consumption needed by the app - Conditional filtering And more. Once you define the requirements appropriately pick relevant services. For example Amazon Eventbridge provides detailed conditional filtering, SNS provides order, etc.

Designing a Pub/Sub system involves steps like identifying requirements, defining topics, setting up publishers, implementing subscribers, handling message delivery, ensuring scalability, managing failure scenarios, monitoring, security, documentation, and testing. By following these steps, you can create a robust system tailored to your project's needs.

The publish-subscribe (pub-sub) model offers a range of benefits that make it an attractive choice for building scalable, flexible, and decoupled systems. Here are some of the key advantages: Decoupling of Producers and Consumers Scalability Enhanced Fault Tolerance Simplified Communication Two Efficient Resource Utilization Simplified Application Logic Real-time Data Processing

Netflix leverages the pub-sub model to enhance its streaming service, benefiting from loose coupling, scalability, and fault tolerance. This architecture allows seamless communication between microservices for user actions like play or pause, enabling real-time updates for analytics, recommendations, and billing without direct dependencies. Pubsub supports Netflix's massive scale by facilitating easy scaling and ensuring the platform remains robust against service failures, thereby maintaining uninterrupted streaming for millions of users globally.

Pub/Sub architecture offers decoupling, scalability, flexibility, asynchrony, fault tolerance, event-driven capabilities, modularity, reduced complexity, reliability, and extensibility, making it ideal for building distributed systems.

It's less of the challenges mentioned, rather solving all of them with a single tool/service. For example, Amazon Eventbridge has archive and replay, which makes retesting much easier, but it doesn't support FIFO. Decide the priorities of the project and select the tools accordingly

Uber's real-time dispatch system, built on a pub-sub architecture, faces challenges like ensuring event ordering for ride matches, securing sensitive data, and debugging in a distributed environment. To tackle these, Uber employs strategies like stateful logic for event consistency, strong authentication and encryption for security, and advanced logging and tracing for debugging. Additionally, Uber uses idempotency checks to handle duplicate events, ensuring reliable and efficient operations despite the complexities of managing a vast, dynamic marketplace of riders and drivers.

Challenges of Pub/Sub include complexity, message ordering, scalability, latency, message duplication, error handling, security, monitoring, interoperability, and message persistence. Addressing these requires careful design and implementation to ensure system reliability and effectiveness.

Google implements pub-sub in its e-commerce platform using Google Cloud Pub/Sub for real-time order processing. When a customer places an order, the system publishes an event to the order-placed topic. Various services subscribe to this and related topics for tasks such as inventory updates, sending notifications, and order fulfillment. Each service processes its respective events independently, enabling a scalable, decoupled architecture that ensures efficient, reliable handling of orders and related processes. This setup exemplifies the use of cloud-based pub-sub systems for complex, real-time workflows in a big tech environment.

To implement the pub-sub pattern, start by selecting a messaging system such as Apache Kafka, RabbitMQ, or a cloud service like AWS SNS. Next, define communication channels or topics where publishers can send messages, and subscribers can listen to relevant information. Publishers are responsible for sending messages to specific topics whenever there is relevant data to share. On the other hand, subscribers express interest in specific topics and receive messages when relevant events occur. It's crucial to design the system with loose coupling between publishers and subscribers, allowing for scalability and flexibility. This ensures that components can be added or modified independently, contributing to a more robust and adaptable system.

Implementing Pub/Sub involves choosing a messaging system like Kafka or RabbitMQ, defining topics, setting up publishers to generate and publish messages to specific topics, implementing subscribers to receive and process messages, configuring message brokers for routing and scalability, ensuring message delivery, scalability, and fault tolerance, handling security concerns, testing system functionality and performance, and documenting the architecture for clarity and maintenance.

Besides the usual suspects that's mentioned here - SNS/SQS/Eventbridge, you can also be nifty and integrate SNS > SQS > EKS, where business logic in Kubernetes is processing the messages, and scaling based on KEDA (Kubernetes Event Driven Autoscaler).

In some cases, you absolutely need synchronous communication. Pub Sub doesn't fit the bill there. However, most systems can be broken down into combination of pub-sub and synchronous communcation

The publish-subscribe architecture in event-driven systems involves publishers sending messages to topics without knowing who the subscribers are. Subscribers listen to topics of interest and receive messages accordingly. This decoupling allows for scalable and flexible systems. For example, in Google Cloud Pub/Sub, a retail company can publish sales events from various locations to a "sales-events" topic. The inventory management system subscribes to this topic to update inventory levels in real-time, demonstrating efficient communication and processing without direct linkage between publishers and subscribers.

Event driven systems enhance the capability of the existing systems to be more decoupled and scalable. Not all systems need event driven architecture, but if you are in a process to break down your monolithic systems and move towards more decoupled or decentralized architecture consider event driven architecture as one of the pattern. If not the entire system, but some of the modules will benefit from this pattern. More benefit comes from implementing the pub/sub pattern for communication between micro services.