Building High-Performance Notification Engine Using Pure Functional Scala, ZIO HTTP, ZIO Kafka. Event-Driven Architecture.
Oluwaseyi Otun
Lead Backend Software Engineer ( Scala, Python, Java, Kafka, ZIO, Akka)
A notification system is an important and integral part of other systems in any organization, whether you’re designing an e-commerce platform, trading platform, or a recruiting system, such as indeed.com, Notification services are widely used in any product. Notifications are critical to increasing user engagement whether you’d like to be notified of a change in price (ideally a price drop) or the availability of a product that you are interested in, or whether you’d like to be notified if there’s a new job specification available for a job search criteria that you have specified. Sending notifications is a requirement of many modern applications. A good notification system must have multiple delivery channels such as Email, SMS, WebHook, Slack, and many other channels.
In this article, we are going to build a scalable Notification Engine using ZIO Stream, ZIO HTTP, and ZIO Kafka using event-driven architecture and a ZIO application layer. This article will require a basic understanding of Kafka, ZIO, Scala, and building an effects system using the ZIO pure function scala library.
What is an Event Notification?
Events are essentially notifications of a change of state of something of interest, something that has happened. They cannot be changed and are immutable. Examples of events include:
Event-Driven Architecture
Event-driven architecture (EDA) is a design pattern in which decoupled applications can asynchronously publish and subscribe to events via an event broker (modern messaging-oriented-middleware) i.e Kafka, Pulsar. Event-driven architecture is a way of building enterprise IT systems that lets information flow between applications, microservices, and connected devices in a real-time manner as events occur throughout your business, instead of periodically polling for updates. It brings speed and agility into everyday processes and boosts efficiency.
What is ZIO
ZIO is a zero-dependency scala library for asynchronous and concurrent programming using non-blocking fibers that never waste or leak resources. ZIO is a good candidate for building scalable, resilient, and reactive modern business applications. To learn more about ZIO check the official documentation and I have written another blog post.
ZIO Kafka
An event-driven architecture is a paradigm that has become increasingly used in modern microservices-based architectures. It promises a more flexible and responsive architecture to business events while offering better technical decoupling. Apache Kafka is an ideal candidate for building an event-driven system. ?At its heart sits a distributed log. What the log-structured approach does is very straightforward. It is basically a collection of messages, appended sequentially to a file. If an application or service wishes to read messages from Kafka it first locates to the position of the last message it read, then scans sequentially, reading messages in order, while periodically recording its new position in the log.
At its core, Kafka has characteristics that set it apart from traditional enterprise message queues:
Kafka client library is used to interact with Kafka broker. It has various programming language implementations such as Scala, Java, Python, Ruby, etc. using this library can be a daunting task for developers implementing patterns like buffering, a batch of records, chunks base processing, parallel processing, etc. ZIO Kafka is a lean and expressive library for interacting with Kafka through a ZIO Streams-based interface with Kafka broker using asynchronous, non-blocking parallel processing that integrates well with ZIO.
ZIO Kafka makes you focus more on business logic and ensure resource safety and manages parallelism for us using ZIO fibers. It can process messages concurrently using ZIO fibers. Learn more about ZIO Kafka
ZIO HTTP
ZIO HTTP?is a powerful library that is used to build highly performant HTTP-based services and clients using functional scala and ZIO and uses?Netty?as its core. ZIO HTTP has a powerful pure functional library that helps in creating, modifying, and composing apps easily. Learn more about ZIO HTTP
Use Case
Lasgidi is a fictitious startup fintech company that provides financial services across North America. They just raised Series A funding from investors. As a startup, they have adopted a functional effect system to rapidly build and scale their solution to millions of users across North America. The notification system is an integral part of the service ecosystem. They want to take advantage of effects systems to build a Notification Engine that is very highly optimized for speed, memory & CPU efficiency, resource safety management, testable, composable, and maintainable, and reason about by design. They have decided to go with?ZIO a pure functional Scala library for building effects systems.
Below are the functional and non-functional requirements:
High-Level Architecture
Notification Events: Event is modeled as a simple case class that has a channel, payload, and key. The delivery channel can be either SMS, Email, or Webhook. The payload is a simple JSON data format that carries the message of the event and has delivery channel property.
Let’s review the code of our Events.
Notification Gateway Publisher: The event API gateway is an entry point to our notification event bus. It exposes an HTTP restful endpoint that clients can call to publish events to the Kafka broker. It has the following components event publisher, notification Service, and ZIO HTTP client adapter.
Event Publisher: It is a ZIO Kafka Producer implementation that produces an event to the Kafka broker. It uses JSON data format to send the event to the Kafka message broker.
Let’s review the code of our?Event Publisher.
Event Notification Service: It is a ZIO Kafka Producer API implementation that produces an event to the Kafka broker. It uses JSON data format to send the event to the Kafka message broker.
Let’s review the code of our Notification Service.
HTTP Client Adapter: It is a wrapper on top ZIO HTTP client library adapter to send single and multiple requests to an HTTP server endpoint.
Let’s review the code of our?HttpClient adapter.
领英推荐
Notification API Gateway.
Let’s review the code of our?NotificationAPIGateway.
Notification Event Bus: Event bus model using ZIO Kafka consumer. It has an event handler that processes the events that are produced by the event publisher to the Kafka messages. The event handler is defined as a higher-order function type in Scala.
Event bus
Let’s review the code of our?EventBus.
Notification Event Handler
Let’s review the code of our?EventHandler
SMS Connector: Connects via Twilio to send sms.
Let’s review the code of our?SMSConnector
Email Connector: connects via SMTP to send emails.
Let’s review the code of our?Email Connector
WebHook Connector: connect via HttpClient and send webhook message
Let’s review the code of our?WebHook Connector
Plugging everything together
In the main program, we wired all the dependencies up and run our effect in the main method.
The full source code can be downloaded on my Github repository. Feel free to clone the repo modify, play around, enhance the source code and adapt it to your use case.
Takeaways
In this article, we learned some important capabilities of the?ZIO effect with ZIO Kafka, ZIO Http which helps us to write high-performance and concurrent applications with?ZIO?fibers using the functional programming paradigm.?ZIO?is a great tool for building resilient and high-performance systems without worrying about deadlock, resource leaks, and race conditions.
Our implementation is very concise and easy to reason about. With a few lines of code, we have implemented a high-performance Notification Engine that is can deliver notifications via various delivery channels such as SMS, Email, and Webhook which is the power of?ZIO?and pure?functional programming(FP). Stay tuned for more?ZIO?blogs.
Special thanks to?John De Goes, Adam Fraser, Wiem Zine Elabidine,? Kit Langton,?and Ziverge team for creating ZIO?and other?ZIO?contributors and also for evangelizing pure functional programming in scala communities around the world.
Read more on?ZIO?Type-safe, composable asynchronous, and concurrent programming for Scala
Thank you for reading.
Oluwaseyi Otun?is an independent consultant focused on backend and data engineering (Scala, ZIO, Akka, Cats Effects, Kafka, Apache Spark, Java). Distributed system enthusiast and passionate and special interest in pure functional programming with a special interest in?ZIO, software architecture, design pattern, microservice, clean and quality code. I love learning internals working on a system and exploring what happens under the cover.
Staff Software Engineer - Walmart Global Tech
2 年One of the best articles I have found explaining how the different ZIO modules can work together. Thank you for the effort you put into this.
Senior Software Engineer - Scala
2 年Found my weekend plans
Senior Engineering Manager
2 年nice work ! Just curious, how much effort was involved in building this notification engine ?