Building High-Performance Notification Engine Using Pure Functional Scala, ZIO HTTP, ZIO Kafka. Event-Driven Architecture.

A notification system is an important and integral part of other systems in any organization, whether you’re designing an e-commerce platform, trading platform, or a recruiting system, such as indeed.com, Notification services are widely used in any product. Notifications are critical to increasing user engagement whether you’d like to be notified of a change in price (ideally a price drop) or the availability of a product that you are interested in, or whether you’d like to be notified if there’s a new job specification available for a job search criteria that you have specified. Sending notifications is a requirement of many modern applications. A good notification system must have multiple delivery channels such as Email, SMS, WebHook, Slack, and many other channels.

In this article, we are going to build a scalable Notification Engine using ZIO Stream, ZIO HTTP, and ZIO Kafka using event-driven architecture and a ZIO application layer. This article will require a basic understanding of Kafka, ZIO, Scala, and building an effects system using the ZIO pure function scala library.

What is an Event Notification?

Events are essentially notifications of a change of state of something of interest, something that has happened. They cannot be changed and are immutable. Examples of events include:

• A change in the stock price in a trading system?
• A rideshare app like Uber notifies the customer that the driver has arrived to pick up the customer
• WebHook notification to integrating partners for status updates for a transaction
• Email Notification to engage the customer

Event-Driven Architecture

Event-driven architecture (EDA) is a design pattern in which decoupled applications can asynchronously publish and subscribe to events via an event broker (modern messaging-oriented-middleware) i.e Kafka, Pulsar. Event-driven architecture is a way of building enterprise IT systems that lets information flow between applications, microservices, and connected devices in a real-time manner as events occur throughout your business, instead of periodically polling for updates. It brings speed and agility into everyday processes and boosts efficiency.

What is ZIO

ZIO is a zero-dependency scala library for asynchronous and concurrent programming using non-blocking fibers that never waste or leak resources. ZIO is a good candidate for building scalable, resilient, and reactive modern business applications. To learn more about ZIO check the official documentation and I have written another blog post.

ZIO Kafka

An event-driven architecture is a paradigm that has become increasingly used in modern microservices-based architectures. It promises a more flexible and responsive architecture to business events while offering better technical decoupling. Apache Kafka is an ideal candidate for building an event-driven system. ?At its heart sits a distributed log. What the log-structured approach does is very straightforward. It is basically a collection of messages, appended sequentially to a file. If an application or service wishes to read messages from Kafka it first locates to the position of the last message it read, then scans sequentially, reading messages in order, while periodically recording its new position in the log.

At its core, Kafka has characteristics that set it apart from traditional enterprise message queues:

• For one, it is a distributed platform, meaning data can be replicated across a cluster of servers for fault tolerance, including geolocation support.
• It offers strong guarantees that messages will be received in the chronological order in which they were published.
• Configurable retention also means Kafka is equally suitable for real-time streaming applications and periodic batch data pipelines
• Kafka’s storage system can efficiently scale to terabytes of data, with a configurable retention period—meaning even when outages that last whole days occur, once service is restored, the event stream proceeds in order.

Kafka client library is used to interact with Kafka broker. It has various programming language implementations such as Scala, Java, Python, Ruby, etc. using this library can be a daunting task for developers implementing patterns like buffering, a batch of records, chunks base processing, parallel processing, etc. ZIO Kafka is a lean and expressive library for interacting with Kafka through a ZIO Streams-based interface with Kafka broker using asynchronous, non-blocking parallel processing that integrates well with ZIO.

ZIO Kafka makes you focus more on business logic and ensure resource safety and manages parallelism for us using ZIO fibers. It can process messages concurrently using ZIO fibers. Learn more about ZIO Kafka

ZIO HTTP

ZIO HTTP?is a powerful library that is used to build highly performant HTTP-based services and clients using functional scala and ZIO and uses?Netty?as its core. ZIO HTTP has a powerful pure functional library that helps in creating, modifying, and composing apps easily. Learn more about ZIO HTTP

Use Case

Lasgidi is a fictitious startup fintech company that provides financial services across North America. They just raised Series A funding from investors. As a startup, they have adopted a functional effect system to rapidly build and scale their solution to millions of users across North America. The notification system is an integral part of the service ecosystem. They want to take advantage of effects systems to build a Notification Engine that is very highly optimized for speed, memory & CPU efficiency, resource safety management, testable, composable, and maintainable, and reason about by design. They have decided to go with?ZIO a pure functional Scala library for building effects systems.

Below are the functional and non-functional requirements:

• The system should be able to send notifications to subscribed consumers.
• The system should be able to support Email, SMS, and Webhook Notifications.
• The system should be highly available.
• It should be scalable, to handle a growing number of subscriptions.
• It process event notification with a maximum level of parallelism taking advantage of the multi-core CPU architecture.

High-Level Architecture

Notification Events: Event is modeled as a simple case class that has a channel, payload, and key. The delivery channel can be either SMS, Email, or Webhook. The payload is a simple JSON data format that carries the message of the event and has delivery channel property.

Let’s review the code of our Events.

Notification Gateway Publisher: The event API gateway is an entry point to our notification event bus. It exposes an HTTP restful endpoint that clients can call to publish events to the Kafka broker. It has the following components event publisher, notification Service, and ZIO HTTP client adapter.

Event Publisher: It is a ZIO Kafka Producer implementation that produces an event to the Kafka broker. It uses JSON data format to send the event to the Kafka message broker.

Let’s review the code of our?Event Publisher.

• In lines 1-3 we define the EventPublisher trait that has a method for publishing messages to the Kafka broker.
• In lines 4-11 we define and implement the EventPublisher trait method that publishes the message to the Kafka broker
• In lines 13-16 we create a companion object that extends ZIO Accessible traits and we define a variable called live which is a function value that converts function to a Layer that depends upon its inputs, It takes ZIO Kafka Producer and String as input and producer EventPublisherLive as output.

Event Notification Service: It is a ZIO Kafka Producer API implementation that produces an event to the Kafka broker. It uses JSON data format to send the event to the Kafka message broker.

Let’s review the code of our Notification Service.

HTTP Client Adapter: It is a wrapper on top ZIO HTTP client library adapter to send single and multiple requests to an HTTP server endpoint.

Let’s review the code of our?HttpClient adapter.

• In lines 1-3 we define the HttpClientAdater trait that has two methods for sending requests. It has single requests and multiple requests.
• In lines 8-25 we define a case class that extends the HttpAdapter Client and implements the two methods in our trait.
• Using ZIO resilence and retry mechanism we implement retry capability in case of failure or connection error. we try using exponential backoff.
• In lines 27-28 we create a companion object that extends ZIO Accessible traits and we define a variable called live which is a function value that converts function to a Layer that depends upon its inputs.

Notification API Gateway.

Let’s review the code of our?NotificationAPIGateway.

Notification Event Bus: Event bus model using ZIO Kafka consumer. It has an event handler that processes the events that are produced by the event publisher to the Kafka messages. The event handler is defined as a higher-order function type in Scala.

• In line 3 we define the function type EventHandler.
• In lines 5-7 we define EventHandler with the handleEvent method and ZIO effect.

Event bus

Let’s review the code of our?EventBus.

• In lines 1-3 we define the Notification EventBus trait that has start methods it takes 3 parameters an event handler function value, no of parallelism, and topic name.
• In line 16 we use ZIO mapZIOPar a curry function that takes two arguments one the number of parallelisms and a function from f: A => B where A is the record and B can be Task or IO or an effect type.
• In line 21, we decode the JSON using the ZIO JSON decoder library and pattern match with left or right in case there is an error if we encounter an error we log the error otherwise we passed the decoded Event message to the event handler function on line 22

Notification Event Handler

Let’s review the code of our?EventHandler

• ZIO has vertical dependencies composition using constructor argument
• In lines 4-36 we define a case class NotificationHandlerLive and implement handle event methods
• We passed in SMSConnector, WebHookConnector, EmailConnector
• In line 11 we get the delivery channel and pattern match on and call Nofication Connector with right pattern
• In lines 37-41 we parse the JSON payload using generic type paramater method

SMS Connector: Connects via Twilio to send sms.

Let’s review the code of our?SMSConnector

• In lines 2-3 we define a trait with a method to send SMS to SMS provider Twillo
• In lines 6- 21 we define SMSConnetorLive case class and implement SMSConnector traits.
• In line 10 we call Twillo Java API to send SMS since this API is a blocking call we shift the call to the blocking thread pool in ZIO runtime. Any blocking API call should be wrapped with ZIO attempt blocking, ZIO automatically shifts this call to be handled by the blocking thread pool.

Email Connector: connects via SMTP to send emails.

Let’s review the code of our?Email Connector

• In lines 1-3 we define a trait with a method to send email
• In lines 5- 26 we define EmailConnetorLive case class and implement EmailConnector traits.
• In line 10 we call Apache Common Email Java API to send an email, since this API is a blocking call we shift the call to the blocking thread pool using ZIO runtime. Any blocking API call should be wrapped with ZIO attempt blocking

WebHook Connector: connect via HttpClient and send webhook message

Let’s review the code of our?WebHook Connector

• In lines 2-3 we define a trait with a method to send SMS to SMS provider Twillo
• In lines 6- 21 we define SMSConnetorLive case class and implement SMSConnector traits.
• In line 10 we call Twillo Java API to send SMS since this API is a blocking call we shift the call to the blocking thread pool using ZIO runtime. Any blocking API call should be wrapped with ZIO attempt blocking

Plugging everything together

In the main program, we wired all the dependencies up and run our effect in the main method.

The full source code can be downloaded on my Github repository. Feel free to clone the repo modify, play around, enhance the source code and adapt it to your use case.

Takeaways

In this article, we learned some important capabilities of the?ZIO effect with ZIO Kafka, ZIO Http which helps us to write high-performance and concurrent applications with?ZIO?fibers using the functional programming paradigm.?ZIO?is a great tool for building resilient and high-performance systems without worrying about deadlock, resource leaks, and race conditions.

Our implementation is very concise and easy to reason about. With a few lines of code, we have implemented a high-performance Notification Engine that is can deliver notifications via various delivery channels such as SMS, Email, and Webhook which is the power of?ZIO?and pure?functional programming(FP). Stay tuned for more?ZIO?blogs.

Special thanks to?John De Goes, Adam Fraser, Wiem Zine Elabidine,? Kit Langton,?and Ziverge team for creating ZIO?and other?ZIO?contributors and also for evangelizing pure functional programming in scala communities around the world.

Read more on?ZIO?Type-safe, composable asynchronous, and concurrent programming for Scala

Thank you for reading.

Oluwaseyi Otun?is an independent consultant focused on backend and data engineering (Scala, ZIO, Akka, Cats Effects, Kafka, Apache Spark, Java). Distributed system enthusiast and passionate and special interest in pure functional programming with a special interest in?ZIO, software architecture, design pattern, microservice, clean and quality code. I love learning internals working on a system and exploring what happens under the cover.