Exploring Camunda 8’s Architecture : Modern Workflow Automation

Managing complex business processes is a significant challenge for organizations of all sizes. Camunda 8 provides a robust solution for designing and automating these processes. This article explores Camunda 8's architecture, highlighting how its distributed nature enables a highly scalable workflow automation solution.

Why you choose a workflow engine in the first place ?

1. Complex business processes with intricate rules and decision logic are ideal candidates for a workflow engine.

2. Workflows that require significant human interaction and involve complex process flows benefit greatly from a dedicated workflow engine.

3. A workflow engine allows for specialization, enabling dedicated?teams to focus solely on process design and implementation.

4. Embedding complex business logic and process flows directly within? ? applications significantly increases application complexity and makes maintenance a considerable challenge due to a lack of clear separation of concerns.

Key facts about Camunda 8:

1. Modern Architecture: Camunda 8 features a completely revamped, cloud-native, and distributed architecture, designed for scalability and resilience.

2. Open Source with Enterprise Support: Camunda is an open-source platform offering both community and enterprise editions. Enterprise editions provide additional features, support, and deployment options.

3. Java-Based: Camunda 8 is primarily built with Java, making it a natural choice for developers with a Java background.

Core components of Camunda 8:

1. Design : Camunda Modeler (For creating BPMN diagrams)
2. Process : Workflow (Defined using BPMN)
3. Engine : Zeebe Cluster (Gateway, Broker)
4. Client: Application or API (For initiating workflows)
5. Executor: Job workers (For executing the workflow tasks and subroutines)
6. Data/Monitoring: Elastic search (For data collection and analyzing the workflow data).

1. Camunda Modeler is the tool used to design your business process workflows. Here, you visually define how your business process will be automated. Your workflow can include the following elements

a. Start and End Events: These define the beginning and completion of a workflow.

b. Tasks

User Tasks: Represent human interaction in the workflow, often requiring a form to be filled.

Service Tasks: Automate interactions with external systems or services.

Other Task Types: Camunda supports various other task types, such as receive tasks, send tasks, and more.

? ? c. Gateways: Control the flow of the process based on conditions. Examples include exclusive gateways (for choosing one path) and? ? parallel gateways (for branching into multiple paths).

? ? d. Events (Intermediate): Model events that occur during the process? execution (e.g., timers, messages).

? ? e. DMN (Decision Model Notation): Used to define complex business? rules that govern the flow of your workflow. This allows you to separate decision logic from the process flow itself, making it easier to manage and update.

?? f. Sequence Flows: Define the order of activities in the workflow.

2. Workflow (or Process): In Camunda, the terms 'workflow' and 'process' are often used interchangeably. A workflow, defined using BPMN (Business Process Model and Notation), is like a blueprint or template for a business process. A process, on the other hand, is a running instance of that workflow.

3. Engine: Zeebe is the core engine of Camunda 8, responsible for executing process instances (workflows). It provides the runtime environment for workflow execution, managing the state of processes, and distributing tasks to job workers. Lets look into architecture of Zeebe cluster and its components in next section.

4. Client : A simple client can be an application (mobile or web) or API that triggers the workflow.

5. Executor(Job workers) : Connects to Zeebe, receives the job, and executes the logic associated with the corresponding service task. This logic could be anything: an API call, a database update, a calculation, etc.

6. Data/Monitoring: Camunda 8 exports workflow data to Elastic search for analysis and monitoring. This integration allows you to gain insights into your processes and visualize them with tools like Grafana.

Zeebe's distributed architecture

Zeebe Cluster: Zeebe is the core workflow engine in Camunda 8, responsible for executing definitions as process instances. It’s designed as a distributed system, not a single process, to ensure scalability and fault tolerance. A Zeebe cluster consists of multiple brokers, each managing one or more partitions. The distributed architecture allows Zeebe to handle a large volume of workflows and continue to operate even if some of the components fail.

Partition: Think of partitions as similar to partitions in distributed systems like Kafka. They are containers for workflow data (process instances). Each partition holds a portion of the overall data and is replicated for redundancy. Crucially, a single workflow instance is always contained within one partition.

Brokers: Zeebe brokers manage partitions and orchestrate workflow execution. A single broker can manage multiple partitions, handling tasks such as distributing workflow instances, replicating partitions for fault tolerance, and coordinating communication between clients and job workers.

Gateway: The Zeebe Gateway is the interface through which clients interact with the Zeebe cluster. It receives client requests and routes them to the appropriate partition. It performs load balancing across brokers at the request routing level, distributing requests to different brokers. The gateway also manages responses to clients, providing information about workflow and task status.

Persistence Layer: Zeebe's persistence layer is built into the engine and uses a distributed log to store all workflow-related information. This includes details about tasks, process instances, and workflow execution history. You do not need to configure a separate database for Camunda's core workflow data. However, if your tasks require interaction with external systems or have application-specific dependencies, you'll need to manage that data in your own persistent store, chosen based on your specific requirements.

Elastic search: While not part of the Zeebe engine, Elastic search is easily integrated with Camunda 8. This integration allows exporting workflow execution data for analysis and visualization, providing valuable insights for business teams.

Zeebe Broker and Workflow Execution in Camunda 8:

Think of your workflow as a piece of code, and Zeebe as the execution environment that scales it. When multiple users trigger a workflow, it's like deploying multiple instances of that code. Zeebe ensures that each process instance is assigned resources (partitions) and runs in isolation from others. Let's explore how this works in a distributed system.

Example: Loan Request Application Workflow

Imagine a loan request workflow executed by multiple users—User ‘A’ and User ‘B’. Here’s how Zeebe handles their requests:

1. Workflow Trigger and Partition Assignment :? When User ‘A’ triggers the workflow, the Zeebe gateway receives the request? and, using a consistent hashing algorithm (or similar), routes it to a specific partition. A Zeebe broker managing that partition then creates a unique process instance ID and assigns the instance to the partition. The same flow occurs for User ‘B’, and their request will likely be routed to a different partition (potentially on a different broker).

2. Process Execution and Task Assignment: Each process instances are executed in their respective partitions independently . Zeebe generates and assigns unique process instance IDs (internally) for internal tracking. For every task within a process instance (workflow), the broker assigns a unique job ID.If the interaction is task specific, the broker manages execution using the associated job ID.

4. Distributed Datas storage: All process execution data and history are stored in Zeebe’s distributed log, ensuring durability, fault tolerance and scalability.

5. Camunda 8 assigns unique task ID and returns to the client or user and based on the task id Gateway clearly knows which partition the task exist.

As we have covered the basics of Camunda 8, Lets look into the development and deployment strategies in our next article.