What is and is NOT Orchestration?
Sana Remekie
CEO Conscia, Thought Leader in Composable Architecture, Omnichannel Personalization, Top 10 Influential Women in Tech, Public Speaker
Introduction to Orchestration in the Composable Landscape
The term ‘orchestration’ has become quite popular in the composable landscape - you hear the term from CMS vendors, content federation vendors, DXP vendors, DXC vendors, Marketing Automation vendors and so on.? As many of the esteemed voices in the composable space have pointed out before, orchestration is one of those loosely used terms that require clarification.? Today, I aim to shed light on this very topic.??
Understanding Orchestration
The word ‘orchestrate’ is a verb and it requires a subject and a predicate to complete if you use it in a sentence.? So, when someone says, we do ‘orchestration’, you should ask, ‘who’ is orchestrating and ‘what’ are you orchestrating.? When we talk about ‘who’, we’re implying that there is an element of a human-in-the-loop (until we get replaced by bots completely, which hopefully won’t be in the near future).? When I talk about Digital Experience Orchestration (DXO), I mean giving digital teams (‘who’) the ability to orchestrate experiences (‘what’).? The human (or AI-powered decision engine or a combination of them both) must decide what the customer should see/hear based on their real-time context.? This implies that you must provide the tools to digital teams to control the experience at an operational level.??
The Technical Side of Experience Orchestration
Now let’s talk about ‘what’ we’re orchestrating, which in our case is the Experience.? In a composable stack, orchestrating an experience requires orchestrating data and business logic housed in various backend services and exposed via APIs.? This is ‘API Orchestration’.
API Orchestration refers to the ability of a system to chain API calls based on dependencies, call API endpoints based on conditional logic and rules (set by humans or AI), and finally, stitch, transform and shape data returned from various APIs in real-time.? For example, in order to show data on the Shopping Cart page, you must call the Cart API and the Inventory API in a logical sequence and stitch together their responses to present products with real-time inventory.? This would be a simple example of ‘experience orchestration’ that requires API orchestration.??
Leveraging Backend Services through API Orchestration
One of the key aspects of real-time API orchestration is that it leverages the full power of the backend services without moving the data from its source - for instance, @Cloudinary offers transformation APIs that are able to optimize and transform the image and video assets in real-time.? So, API orchestration gives you unhindered access to both the business logic as well as the data delivered by a backend service such as Cloudinary.? Another example would be fetching promotions from TalonOne.? This requires you to provide context and TalonOne evaluates a bunch of rules to determine what promotion is valid in a particular context.? Without leveraging the full power of the backend APIs, we’re simply treating the backend as dumb storage.? To drive this home, consider storing these images and videos in an S3 bucket and accessing these via an API.? That would be considered ‘dumb storage’.
API-first Data Layer for Your Backend Systems
There are times when your backend systems don’t offer APIs.? If you need data from these systems to be made available to a real-time experience, you don’t have a choice but to move the data out of those systems into another API-enabled data repository.? This could be a search index, an API-enabled data cache or another operational data repository.? Vendors offering this capability are the likes of Occtoo, Enterspeed, Netlify Connect and even the DX Graph module of Conscia’s DXO.? All these products allow you to access data within a data repository via sophisticated REST or GraphQL APIs so that you can combine/stitch data in a way that wouldn’t have otherwise been possible.
领英推荐
Data Preparation for Frontend Consumption
Sometimes, the data in your source systems, especially systems of record such as ERPs and legacy databases, is not ready for consumption by the frontend.? For instance, product data may not have the right ecommerce categorization/taxonomy or the attributes are not standardized.? Sometimes data attributes are missing altogether and you want to make sure that products/content records that don’t have complete metadata don’t make it to the frontend.? If you need to validate and enrich your data to prepare for your experiences, it is much more practical to do this as an offline process rather than doing so with real-time API orchestration.? Doing so in real-time would be very problematic due to computational and performance reasons.? Some of the vendors, in particular, Conscia’s DX Graph (shameless plug!) offers data validation and enrichment as part of the API-first data layer that sits on top of legacy systems.
Challenges of Non-Real-Time Data Sync
One drawback of this approach is that it doesn’t access time-sensitive data from the backend in real-time.? For example, if you want real-time inventory, syncing the data into another repository or index is not a great idea.? You also can’t go fetch your cart contents using the Cart API from your commerce engine in real-time and use that to determine the next course of action.? On a side note, I can’t help but ask the question - why would you ever sync your data from headless CMSes and other scalable headless services into another operational repository?? Don’t they already have CDNs that guarantee performance?? Also, don’t they already have restful and GraphQL APIs?? Some of the vendors in this space even pull in data from headless CMSs who already guarantee 50ms response times!? This is a topic for another conversation.??
Regardless, although this is a valid architectural pattern to make your data ready to be consumed via an API, it is not ‘API Orchestration’.? In these types of systems, you don’t have the ability to sequence API calls or manage conditional logic to deliver contextual data to the frontend. You're simply making data ready to be orchestrated by providing an API layer to your data.
Is Content Federation the Same as API Orchestration?
Now let’s talk about CMS vendors that have the ability to connect to external systems and deliver data to the frontend by making API calls to these systems in real-time.? Many popular headless CMSs offer this capability.? Some call it ‘remote fields’, others call it ‘external references’ or ‘custom fields’.? Basically, the external data is stored as a reference and can be fetched at query time using GraphQL APIs.? This approach is called ‘Content Federation’ by some CMS vendors.? Although there are elements of ‘API Orchestration’ in this approach, the most obvious of them being the ability to call an external API in real-time to fetch data, there are two elements that this approach are missing: 1) Making these calls conditionally based on customer’s real-time context.? 2) Chaining these calls in a certain sequence based on dependencies between APIs.
Looking for the Middle Layer in your Composable Stack?
Having a middle layer between your frontends and your backend definitely is the right idea, but the approach you choose should be based on your use cases.? If you have a legacy system, having an API-first Data Layer would allow you to build modern frontends on top of legacy systems without a migration.? If your backend systems offer headless APIs, real-time API orchestration is sufficient.? All of the vendors in this category are trying to simplify the composable stack and accelerate implementation of composable experiences by providing a single API for the frontends to connect to instead of having to connect to individual backend APIs.
There are times when you may need a combination of the approaches I have described above as they are actually complementary to each other.? This is why Conscia’s DXO offers both an API-First Data Layer to accommodate legacy systems (DX Graph) and Real-time API Orchestration (DX Engine) to cover a variety of customer use cases.? If you have a data layer from another vendor, you can still use the DX Engine for API orchestration and vice versa.? It’s an exciting space and I think it’s becoming quite clear that there will be a huge amount of innovation to make composability work for organizations in the upcoming weeks and months!
One of the main differentiators here is that some orchestration platforms actually store a copy of the data they are processing. Others do some transformation/joining of the data but don't store it in a cache. To me, the actual offering is completely different depending on how data is being processed and/or stored, and what the developer experience is going to be like. It's also interesting how some orchestration platforms aren't really API driven as they are tied directly to a front-end SDK. Certainly lots of different things happening as folks are solving specific business problems and then bringing platforms to market.
Co-founder & CTO @ Alokai | Modernizing eCommerce frontend-first, without risk | MACH Tech Council Member | Forbes 30 under 30 | YC Alumni (W21) | Vue.js CMTY Partner
1 年It's a much deeper dive than my today's post ?? Thanks for sharing! I suppose more content like this has to be developed by people from our industry. New terms are greatly overused, everyone claims they "own it," and people are getting lost. I appreciate a lot your efforts to educate the space! MACH and Composable are already complicated enough, and we need simpler ways of explaining complex stuff. Here's my take on how to differentiate what is and what isn't an orchestration - API Orchestration is a term describing workflow where one API call uses a result of another one as a parameter -API Integration / Data Federation is generally a term referring to the concatenation of multiple API's into one
Content Engineering Consultant @ Albert Heijn | Owner @ The Content Engineering Agency | Data, Digital & Tech
1 年Excellent points Sana Remekie ! I would also like to add that it's not just the user's context and content federation that could enhance the orchestrated experience, but also the context between backends as well. All that stitching of logic in a CMS, or worse, in the frontend, could be reduced by making meaningful connections between backends in a graph (the middle layer you mentioned). Connecting data and content close to its storage level is a very common practice in knowledge management and enables smarter decisioning for a well orchestrated digital experience to handle real-time user's context faster and more accurate (API orchestration's ambition) across several domains (content federation's ambition). But it takes considerable time and effort to do this properly, so I understand all the 'short cuts' vendors promote to achieve parts of orchestration and sell it as the sum of orchestration while never achieving the whole of orchestration we all dream of...
Sales Channels Solution and business Architect
1 年I was just reading this similar post (https://shorturl.at/bkqwB), and then yours popped up! Chuffed to see this chat picking up steam!
Content Architect | Strategist | Evangelist
1 年Good points, Sana Remekie -- especially about "dumb" backends that many data and content stores are. I see orchestration driven by the user's context and behavior, which happens at the front end (Many vendors and firms mistakenly rely mostly on historical CDP data rather than real-time data from the customer's browser, which is why you get recommendations to buy products you already have bought). While there need to be multiple layers of transformations happening as more and more systems and repositories are introduced, the key driver of orchestration is generally some kind of personalization experience. I divide personalization into matching and fetching approaches. Realistically, supporting this personalization needs to be baked in from the start -- it isn't something that can be bolted on later. See https://kontent.ai/blog/orchestration-in-personalization-matching-vs-directing/