So you think you want an integration
The ability to connect systems is greater than ever
On the evolutionary scale of software development we are well beyond climbing down from the trees and making spears out of sticks and rocks. Modern systems have advanced connectivity tools that allow them to communicate with other systems and services. A common standard language has started to take shape, and demand is drawing out an increasing number of utilities and services to automate processes.
And yet, there is so much confusion about integration, that strategists and decision-makers find themselves unable to formulate a strategy, let alone a plan, for such integrations.
What about security and compliance?
On top of this confusion, the gold-rush towards integration (and automation) has trenched up fear and loathing from security and compliance experts across many industries.
For example, if you connect a manufacturing tracking system to hospital supply system, what are the HIPAA considerations? If your newly integrated system is used both in the US, and in Europe, how will the EU web compliance laws affect the transmission and storage of the data? If you use a third party software along any of the data-route, how can you be sure they are compliant? And more urgently, how will an auditor interpret these questions, in the absence of clear regulations and standards?
If your blood pressure went up reading those questions, you are not alone. The competing business demands and compliance necessities have vapor-locked a great many organizations, ensuring that their default action is 'wait and see'.
Wait and see is the destroyer of empires. The most widely used example is Blockbuster Video. As Netflix innovated and responded to consumer demand, Blockbuster flinched. By the time they tried to pivot, stores were being shuttered.
In my experience, this anecdotal tale of failure is rather un-moving to the people who's job it is to safeguard systems, data, and profit. All they want to know is: How will your project ensure that this will not increase exposure, or flag the organization for a security or compliance audit, potentially costing millions?
Well... yeah. I mean, do you just risk it for the biscuit? Or do you let a couple of other people run through that field first to make sure the land-mines are cleared? The basic risk-reward analysis is tough to do if you don't understand one (or both) of the variables.
And that's the key, isn't it. What is the potential reward for connectivity? And how do you answer the risk questions without becoming a systems architect and security expert, while somehow still doing the job you are in now?
I feel that this is a good place to put in a plug for my colleagues. You obviously have to pay the people who do understand these things, and then you have to listen to them. (end of plug).
Computer languages evolve just like spoken languages
A key fact that has transformed software over the past 50 years or so is that languages evolve. I like to use the analogy of the fast-food menu. When I worked at McDonald's (hint: Reagan was still president), a person would order a Big Mac, A large Fry, and A Medium Coke. That was clear, it was accurate and it was easy enough to understand. But it was a lot of words. Fast forward a few years and bing! You order a number one. (Disregard the fact that you still have to answer what size and what drink... the metaphor isn't perfect).
It sounds overly simple, but language evolves the exact same way. A process which is done over and over will eventually develop a handle or a shortcut. These shortcuts get bundled together in different series to communicate complicated, but understood, ideas.
If you tell someone to "make coffee", you don't say something like: "Turn on the cold water and hold the carafe under it until the water reaches that line. Then turn off the cold water. Now walk over to the coffee maker and open the top hatch, exposing the water reservoir..." Every sub-routine, such as fill the pot, or scoop out the coffee into a filter, had to be understood at least once for the command "make coffee" to be meaningful.
Once processes are given a name, the person writing software simply has to type 'make coffee' and all of the previously written code is included. As more and more routines get bundled together, the power of the programmer is increased by having libraries of routines which can be strung together to create new processes (make decaff coffee, make tea, make a triple non-fat almond-milk peppermint latte, you get the idea).
The other way language evolves is standardization. If you want to drink a Coke (never mind how many different types of Coke their are, that's a different rant), you can simply ask for a Coke. But your server needs to be able to understand when someone asks for a soda, a pop, a cola, a soft-drink, etc. The lack of standardization in terms makes integration much more complicated. But today, even as standards branch into multiple options, the time-to-adoption of a standard is accelerating quickly
API standards are emerging, creating more opportunity
Because API standards have sprung up, more and more programs are offering standard libraries of connection points that allow for integration in a predictable pattern. This is the gasoline on the fire that is the mad-rush towards integration. And the newest iteration of API options has lifted integration exponentially.
GraphQL (and associated variations) allows API calls to 'query' an endpoint, not only for the primary object associated with it, but with related objects. It does this in a similar way to query languages such as SQL for databases.
To put it in layman's terms, traditional API's have pre-made response packages. You ask for a number one and you get a Big Mac, Fries, and a Coke. Or you ask for a menu and it gives you a menu. But special orders do upset us. With modern API's you can ask for exactly (and only) what you want (hold the pickle hold the lettuce and make that a vanilla milkshake). This is overly simplified, of course. But the point is, that the newer API endpoints give a much greater and more efficient way for external programs to get to the information they need.
On top of these standardizations there are dozens of low-code / no-code platforms popping up which help to shorten the lead time even further with ready-made connectors for many popular programs.
It's complicated...
Of course, everything comes at a cost. With more options, and more complexity, comes a greater need for specialists. It's not enough these days for an integrations engineer to understand HOW to get the data she needs. She also has to understand how all of the data is related, both in terms of data-relationship, and in the business sense. That's difficult enough, when you're working with only one system all the time. But it becomes much more time-consuming when connecting this system to something she has never touched, or seen, or been involved with. Such is the case with every custom integration. For every hour coding there are many hours of research.
Here's an example: A client wants to sync their Salesforce instance to a fulfillment system, which tracks the flow of orders. They want a full, two-way sync of customers, orders, and sku's, and they want real-time inventory levels to be available from within Salesforce, as well as regular notifications at specific stages of shipment. Easy-peasy.
领英推荐
All she needs to do now is quickly understand the complete fulfillment process, as well as how inventory is done, as well as what custom setup has been done in SalesForce, and what notifications are currently being sent from which system, and which ones do they want to keep, and which ones will be migrated, and who will get these notifications, and what are the use-cases for each one, and how do orders flow now, and who should have access to what, and... (simple right?).
In the above example, the client wants to skip ahead to full integration of several entities in a single integration. It can be (and has been) done, of course. This common one-and-done integration rarely gets done on-time and on-budget, and often causes unintended consequences not considered until they rear up and bite you on the proverbial arse.
A more iterative approach could actually provide the client with a completed, less buggy, system in a much shorter time. Less is often more, when it comes to integration.
It's going to take longer and cost more than you think
I know some of you are scratching your head and saying, "This guy sounds like the idiot I hired to do our integration." And there is a reason for that.
Less is more
Less is often more when it comes to a data-sync. Two-way sync's are inherently tricky because you have to prevent looping updates. When an update in either system triggers an update in the other, the second update has to be stopped from calling a third (and so-on).
There is a principle in data design that says you probably don't want to put the same information in more than one place, because it's more storage, more maintenance, and more opportunity to be wrong. With integrations this is sometimes unavoidable. A "Source of Truth" is necessary for every entity stored across platforms. For example: Salesforce is the source of truth for shipping address, so it never gets updated directly by the sync, but requires an explicit update. If they are out of sync, you trust the source of truth to be correct.
The complexity of updating multiple processes at the same time also makes it much harder to determine which one caused the inevitable snafu. It's like having six kids and finding the broken lamp in the family room... you can't just punish Peter, you have to be sure it wasn't Marsha, Cindy, or that other girl.
Hidden work-around's and non-standard flows
On top of that fact, there are always non-standard work flows hidden away that do not fit neatly into every box. From a data-layer perspective, business processes that aren't part of the (software) system never happened. (If it's shipping to College Station, the warehouse substitute one of the maroon stickers instead of the standard orange ones.)
Even within custom software there are often hard-coded surprises waiting that fixed some obscure problem, but have not been viewed or thought about since that developer left for a job that let him work from home. When a newly standardized sync hits these little joy-bubbles it can be a nightmare figuring out why, for example: a blue, XXL hoodie is listed as a spatula if it's the second Thursday in July.
Talking through all of these processes is expensive, time-consuming, and often frustrating to the managers who had a hard enough time selling this integration to their bosses when they didn't understand how hard it would be. Going back for more time and money is going to hurt. The most important work done during an integration happens in the planning stages.
Diagram your current workflows
Before you bring in an outsider, or hire a new integrations engineer, make sure that you actually understand how your system works. It sounds obvious, and maybe even a bit sarcastic, but it's going to save you time, money, and heartburn.
Diagram out the flows with your key players and managers until there is concencus. You will likely be surprised how many efficiencies you might come up with just by taking the time to do this. But the real value of putting them in writing will be the ability to share them with your integrations team.
Pro-tip: Mama always said that family doesn't have squabbles in front of company. Maybe have these discussions before you include the consultants and engineers in your meeting.
The data map is the golden ticket
When an integration is well planned, there is a document(s) worth it's weight in bitcoin: the data map. Determining which entities map to which other entities in the connected systems, and know where they are stored, what they are called, and what they are associated with in every system is vitally important.
When using more than two systems, it's a good idea to settle on a neutral, easily understood description for each datapoint. If you rush through this step, or if you don't involve the data experts from each connected system, you are going to have a project hangover that rivals anything Charlie Sheen has ever experienced.
Integration is a game-changer. The efficiencies it can create can reduce costs, improve time-to-value, and free up humans to do more productive tasks. It requires planning and collaboration to be done correctly. If it's not done correctly, it can add surround sound to your most vivid nightmares. Proceed with caution.
Key steps:
John is an Integration Engineer. He likes short walks on the beach, loves dogs and old trucks, and enjoys cigars and tequila. His opinions are his own, and do not necessarily represent the views and opinions of any other person (living or otherwise), his company, or anyone else. He encourages you to have your own opinions and will proudly support your right to do so, though he will mute notifications from you if you're obnoxious about it.
Connector | Agilist | Certified Scrum Master | Certified Scrum Product Owner | Project Management Professional | Learn-It-All | Heart Centered Leader
2 年John S. This is an incredible article and resource! My favorite line “Pro-tip: Mama always said that family doesn't have squabbles in front of company.” I’m saving this to share as applicable. Thank you dear friend!!