Part 3: The Sync Engine

Intro

Part 1: The Datastore

Part 2: Connectors

Part 3: The Sync Engine

Part 4: The Workflow Engine

Part 5: The Customers and What to deploy?

Part 6: Engineering challenges - Connectors

Part 7: Consulting challenges – Why do most RBAC / ABAC Deployments Fail?

Part 8: IAM People Management challenges

State Machines vs. Event-Driven Systems

To start, let's clarify the concepts of state machines and event-driven systems in the context of Identity Management (IdM).

State machines

A prime example of a state machine is the Microsoft Identity Integration Server (MIIS) / Forefront Identity Manager (FIM) / Microsoft Identity Manager (MIM) Sync Engine. This engine represents the state of a person object in the metaverse and applies provisioning and synchronization rules to create or update account objects in a target connector space, generating an "as-to-be" state hologram. By maintaining a hologram of the imported object or "as-is" state, MIM Sync constructs a delta to address any discrepancies. This approach also introduces near real-time reconciliation, ensuring consistency regardless of the reason for discrepancies in the target object. For instance, it could be due to a new job title imported from an HR system or an unauthorized manual update by an admin in the connected system. The same applies to group memberships; MIM will simply roll back any unauthorized changes to groups if no equal precedence for group members is defined for that connector. I’ve seen admins trying to fight with MIM Sync by manually adding users back into AD DS groups managed by MIM – with no luck at all.

Pros:

·???????? Aggregates changes.

·???????? Built-in data reconciliation.

·???????? Rollback of unwanted changes.

·???????? Thresholds and export queue control.

·???????? Preview mode.

·???????? Reports of pending changes.

·???????? Up-to-date references.

Cons:

·???????? Not real-time.

·???????? Difficult to scale horizontally.

Event-Driven Systems

Entra ID Cloud Sync Fabric exemplifies an event-driven system. In such systems, an incoming change (via a message bus or a trigger) from HR prompts an update to an object in a connected system. However, without a current state representation of an object, there are two options: read the object prior to making an update to build a delta (slow by design) or replace the entire object (ineffective for group members). Consequently, unauthorized changes might be ignored, or the export process will be slow. Another issue with event-driven systems is managing references, as a new employee may be imported from HR minutes before their new manager, necessitating initial provisioning without a manager and subsequent reference updates. It’s not uncommon to see systems that require group owners to be present for new groups and such requests failing because of a missing manager (that is imported a bit later).

Pros:

·???????? Scalability with multiple workers listening for specific topics.

·???????? Concurrent export.

·???????? No need to manage holograms.

·???????? Easier implementation.

·???????? Supports export queues.

Cons:

·???????? Slow if reconciliation is needed; otherwise, no reconciliation.

·???????? Poor handling of out-of-order changes.

·???????? Inefficient reference handling.

·???????? Lack of change aggregation, leading to multiple updates to the same object

Combining Both Approaches

Combining state machines and event-driven systems is feasible, albeit time-consuming and budget-intensive. The goal is to implement a proper sync engine that leverages the strengths of both approaches.

Synchronization and Provisioning Rules

From experience, it's advisable to avoid custom project / join / resolve / provision / sync rule extensions and external calls from sync rules. Properly scoped sync rules eliminate the need for these extensions. The declarative rules in MIM Sync and Entra ID Connect Classic are sufficient for most Identity Management cases. Scoped declarative sync rules with basic property manipulation functions should suffice. Sync rules should include join / filter / project / provision criteria for all connectors and attribute flow precedence in the metaverse.

Technical Implementation

A modern sync engine could involve worker processes listening to message bus events (e.g., Pending Import Hologram updated), creating a queue of objects for synchronization, applying sync rules per object, updating references in export holograms, and staging holograms for export. This mirrors the MIM Sync's delta sync process but in a modernized manner.

Incoming "update pending-import-hologram" messages should not be marked as processed until sync rules are applied and export holograms updated.

A little bit on Dynamic Group Membership and Sets

Maintaining relationships between accounts (not persons) and roles/groups in a graph database is more straightforward than in a traditional SQL database with normalized data. Dynamic criteria for sets/groups can be managed by tracking updates to person properties and updating sets/groups accordingly. However, this asynchronous approach can introduce race conditions for dynamic group membership calculations, similar to those experienced in MIM Service when multiple properties are updated simultaneously.

The simplest solution to this issue is to eliminate support for nested sets and groups, as seen in Entra ID. Without nested structures, handling memberships becomes more straightforward, albeit less flexible.

Another challenge with sync rules, policies, and dynamic membership in MIM was related to the 'before' and 'after' sets of target objects. To decide whether to apply a policy (and initiate a workflow), MIM had to calculate all potential transitions IN and OUT of all sets before committing a request to MIM Service database (or denying it). This time-consuming procedure does not scale well, though it provides flexibility in designing policies and actions. Validation of these transitions could be offloaded to worker processes to improve scalability.

By leveraging modern technologies and combining the strengths of both state machines and event-driven systems, a contemporary sync engine can provide efficient and scalable Identity Management solutions. Balancing the flexibility of dynamic group memberships with the need for performance and reliability remains a critical consideration in the design of such systems.

Previous: Part 2: Connectors

Next: Part 4: The Workflow Engine