S01E04 dapr-compass - No actor is an island

If you missed the first three articles of the series, here’re the links:

• S01E01 dapr-compass: how the story begins
• S01E02 dapr-compass - how we organized the work
• S01E03 dapr-compass – The Front End API interface

Let’s first quickly recap the use case we chose for our Dapr adventure.

In a retail organization it’s quite common to have multiple sales channels ultimately offering a shared set of products: you can buy a pair of flip-flops via mobile app or in store, tapping on a kiosk or in a mall, the shoes then could be put in a bag and given to you, sent to your home or you can pick them up at the nearest store (maybe trying them on, as this is normally a recommended pattern, for shoes and flip-flops anyway).

Large organizations provide their customers with multi-channel experiences: offering many brands while the online/mobile eCommerce page is tailored for the specific audience or by gathering social media associated insights for that brand.

In the end, every single channel will need to check the availability of the flip-flops in the nearest warehouse, and possibly reserve the last pair before Karen gets there first. A trivial task, it seems, until you consider the massive scale these channels can serve, and the heterogeneous access patterns.

The combined demand can become too high for the ERP not to suffer, and shifting the burden on each retail subsystem, on the other hand, can lead to over-complicated sync processes.

It is therefore a common practice to dedicate a system to managing product availability, registering reservations and imposing automated expiration if there's no commit into a final order. Few simple tasks that each retail channel can leverage, counting on a high performing architecture that bears high load with highly concurrent interactions.

Back to today’s topic, as soon as we started to put our head into the task, a tough question popped to our minds: 2orchestrate|!2orchestrate?

Actors are the smallest computation units in Dapr, which can run concurrently, independently and in large numbers.

The Dapr documentation on Actor model tells us that “an actor is an isolated, independent unit of compute and state with single-threaded execution. Dapr provides an actor implementation based on the Virtual Actor pattern which provides a single-threaded programming model and where actors are garbage collected when not in use. […] A single actor instance cannot process more than one request at a time. An actor instance can cause a throughput bottleneck if it is expected to handle concurrent requests. Actors can deadlock on each other if there is a circular request between two actors while an external request is made to one of the actors simultaneously. The Dapr actor runtime automatically times out on actor calls and throw an exception to the caller to interrupt possible deadlock situations.”

Working on the StockActor type and with the initial requirement of publishing changes via messages, we started discussing the general approach: is it worth the risk to establish dependencies between actors? Or should we keep each one independent, possibly orchestrated by an external service?

In the simplest case, the client intent is expressed by interacting with the actor, which will be fulfilled only if the action on the depending actor is successful, subsequently publishing data to other subscribers. The obvious PRO: it’s a lot simpler to develop, the CON is that the actor does implicitly take an orchestrator role.

In the following diagram, a customer interacting with a website is booking a pair of shoes: #1 the public API recognizes the intent, invokes the (new) OrderActor instance which accepts the request only if #2 the StockActor instance for the corresponding SKU successfully allocates the needed quantity. Before acknowledging the success #3 the OrderActor publishes the message to be consumed by other systems.

Alternatively, a microservice could be the orchestrator of all the actors calls and taking care of the publish/subscribe actions. PRO: each actor will operate without dependencies or added I/O, limited by only its capacity. CON: adding a layer will make the development a bit more complex, and also, as we animatedly discussed, this might not make any sense at all in the overall big picture of Dapr and Kubernetes, and against the definition of independency of the actors model.

In the diagram below #1, #2 and #3 are orchestrated by a dedicated service, with no direct communication between Order and Stock actors.

Bear in mind that the external API, a node.js service which Dapr injects with env variables to compose urls, has no SDK or libraries: in order to demonstrate how a service with no dependencies to Dapr can benefit from it, we chose this path. Otherwise the API would be a potential scope to take the orchestration responsibility.

A good reason to implement Order as an actor is to leverage the Reminder feature, see documentation for more details.

Timers in Dapr enable you to “register a callback on actor to be executed based on a timer. The Dapr actor runtime ensures that the callback methods respect the turn-based concurrency guarantees. This means that no other actor methods or timer/reminder callbacks will be in progress until this callback completes execution.” And while timers are stopped when the actor is deactivated, Reminders let you trigger persistent callbacks on an actor at specified times.

In our dapr-compass solution it is important that a #1 pending reservation, after a period of inactivity #2 by the client, is automatically cancelled by #3 compensating the available quantity.

A diminishing factor of risk is the client audience for each actor: Order actors tend to have just one client, the customer invoking the API from one of the many retail channels, which then waits for the response. OrderActor also has a shorter lifespan, in an ever-increasing population of instances.

Our StockActor type, instead, is the central information, a long-lasting smaller population of instances, receiving many concurrent requests for which Dapr will provide protection via serialized access.

Considering the Dapr and Virtual Actor recommendations, this seemed a minor infraction: we decided for the simplest option as the Order actor directly invokes the Stock actor.

We will keep this decision in mind during the load testing phase: in case there are unpredictable consequences, we will rediscuss the scenario and change our approach before going into production ??.