Integrating Cloud based SaaS applications

Overview:

Industry is moving towards Cloud technology. A lot of work that was done on-premises is now migrating to the Cloud and so are various SaaS applications.

The homogenization of these applications in the Cloud not only provides tremendous value to the industry but also increases productivity and data quality.?Integrating various SaaS applications in the Cloud is a tough journey with various challenges but certainly not impossible if the proper approach is taken. As it’s said, “All good things are difficult to achieve and it’s in moments of decision that destiny is shaped”.

Integration Journey:

What is the need?

To wisely design an integrated solution, we need to understand what the need is to integrate. Two main reasons to build integrations between applications are to improve productivity by saving significant amount of time in data entry and to improve data quality across these applications. Better data quality means better output, better reporting, and an increase in business.

What are various categories of data?

Generally, applications need three major types of data – reference data, master data and transactional data. Reference data can be understood as static data set that supports business processes such as job profiles, provider status, site status etc. in your organization. Master data supports transactions like employee data, site data etc. ?Transactional data describes different events on master data and is the largest volume of data in an organization.

The Journey

Integrating live applications is challenging and requires thorough prep work in terms of understanding different data elements and existing processes. ?This journey will also necessitate instituting a data governance program to ensure data quality and accuracy.

After the integration, before things start syncing, it can feel like everything broke and the integration is a disaster.?This is very normal. ?Again, data governance plays a very important role in data cleanup. Post integration cleanup can be minimized by correct project planning and approach used in integration.

While dealing with SaaS solutions, we need to recognize the fact that these SaaS solutions will undergo upgrades, which can break integrations from time to time. A proper change control in place minimizes this issue.

Another thing to keep in mind is that each of these applications has their own business process and timing to process the data to support their operations. Understanding and scoping these while designing the integration solution is very important. For example, a worker terminated in one system may still be needed as an active employee in another system for some time to support business processes.

What we are dealing with here are systems that are in the Cloud and most often are multi-tenant, which can mean the data model and schema underneath them is very proprietary. When fetching data from one system and sending it to another system, it becomes very important to understand the data model that exists in these systems. To address this complication, building a master data model that is specific to the organization's need is of great value. And the goal of integration is to conform SaaS data models to your organization's specific master and relationship data model.

A big decision to be made is the data synchronization approach between systems and timing on data updates from source to destination. Destination systems may only need a subset of master and reference data, so updates should send only the required data elements. ?If filtering logic is not in place and data is refreshed at destination with any update on source systems, then audit logs will be cluttered. ?Therefore, the decision needs to be made with proper consideration.

It is also essential to consider the timing of when the destination systems need the data sync, i.e., real time v/s on schedule or on-demand. This plays an important role in designing the architecture of the solution.

Choosing best technology

The Big Question is what technology is the best to build Cloud integrations.?There are a good number of old and new technologies such as Microsoft Integration Services, Informatica, Mule Soft, IBM App Connect, AWS Application Integration, etc. in the marketplace.?The selection of technology depends on several factors such as the technology stack and resource skill set at your organization, strategic direction of your enterprise framework, heavy code vs low code approach, as well as other factors that are unique to your organization.?Select the one that is most suitable for five years or more. Below are some points to consider when making decisions on technology and building architecture:

Data mapping

Data mapping will be done in the tools and technology used for integration; however, to have a clear understanding amongst business stakeholders, architects, developers, and testers, a much simpler data mapping document should be established and maintained in an Excel spreadsheet or similar tool. Make sure this document also has version control to understand changing data mapping needs.

Batch processing for data recovery

Always account for system failures.?Systems, including integration processes, will fail sometime, and that is a reality.?Build integrations to recover from failure, for example batch synchronizing a subset of data that failed due to system failures or outages.

Error Handling

Build a proper error handling and logging framework.?It is critical to troubleshoot if anything goes wrong.?Use existing technology components to define error handling and logging framework that is specific to your organization, such as integrating with a ticket management system to auto create tickets to troubleshoot and fix issues.?Use extensible technology components such as NoSQL tables to capture ever-changing needs of capturing errors and logs.

Publisher-Subscriber model

As a rule of good architecture, do not build point to point integrations.?Use Publish-Subscribe framework and components like service bus, events, etc. to build asynchronous hub and spoke integration framework.

Conclusion:

To stay competitive, organizations need good quality information from good quality data that is available on-demand and in a timely fashion.?Integration not only brings systems in sync but also makes data complete.?The need for integration is clear and the availability of technologies to achieve this are plentiful.?However, the path to get there has some challenges.?With proper consideration and planning it is totally achievable.?There will be hurdles initially but the reward at the end is promising.?

Additional Contributors:

Aditi Jain

Joanne Krogman