Achieve Jitterbit Reuse

How do I achieve Reusability in Jitterbit implementations?

A favorite topic that I never tire of researching. Arguably reuse is the secret sauce to creating repeatable, maintainable and robust solutions

The following are concepts for reuse, if they are new to you, don't get disheartened, just acknowledge the principles for later examination:

• Modular programming is a technique that emphasizes separating the functionality of a program (re projects) into independent, interchangeable modules, such that each contains everything necessary to execute only one aspect of the desired functionality
• Coupling refers to the degree of direct knowledge that one component (re operation) has of another
• Single responsibility principle states an object (re operation) has the responsibility over a single part of functionality

To get the most out of this article, please familiarise yourself with Jitterbit Tips & Tricks

Context

In order to frame this discussion, let us consider an ETL use case, modelled in a Citizen style and then modelled in a Developer style practicing reusability

Use case: Extract data from Salesforce, Transform, Load to Magneto and Update Salesforce with Magneto Id. (Noting Magneto has no bulk upsert for arguments sake)

The following are illustrations to emphasize concepts and are not ideal solutions

-----------------------------------

Citizen Style

-----------------------------------

Beautiful, simple and works for the happy path

• But what if you need another Salesforce Contacts Query or Salesforce Contacts Update?

1. We can not reuse these as they are locked (highly coupled) to the process of Magneto Customers
2. Create another Salesforce Contact query and maintain two? Salesforce interface changes it will now impact two operations
3. Take the route of duplication, but what if the way you handle Salesforce errors changes?

• But what if you want to import the hard earned Magneto knowledge to another project? It will drag Salesforce objects with it!
• But what if you need to insert Magneto customers from another non Salesforce contacts source?

1. Even if you try and squash non Salesforce data through 2 - Transform and Insert Customers operation the on Success event fires a Salesforce Contact Update in a highly coupled solution

• But what if 3 - Update Contact with MagId throws an error do you think 1 - Query Contacts will be alerted to the error? No it will consider 2 - Transform and Insert Magneto Customers ran successfully
• But what if volumes increase from Salesforce and you want to add parallel processing / chunking to the process. Enable chunking is not available on 1 - Query Contacts

-----------------------------------

Programmer Style

-----------------------------------

Computer Says No! Run for the hills, I'm scared.

But wait, the following is what it looks like with the Lego/reuse operations removed

Modular Design

We have plugged in generic Salesforce & Magneto operations who are only concerned at being the best ever Salesforce/Magneto operations, their responsibility is to make sure they are bullet proof in their conduit role to an endpoint and they are not concerned on who invokes/runs/calls them as long as their interfaces are satisfied.

Now there is clear separation of concerns:

• Orchestration - Allows the decoupling and graceful error handling. Tip 6 - Orchestrate To Success and see Project Orchestration Best Practices Tech Talk
• Extract - A reusable Salesforce query with the SOQL passed in (Tip 3 - Global Variables) allows this operation to be concerned only with fetching the data, nothing else
• Transform - Now the majority of the logic, hard work and bugs of an integration can be found here

1. You will have captured runtime data using Tip 8 - WriteToLog and therefore easily debug/develop using Tip 9 - Design Time Development
2. Opportunity to introduce a canonical model
3. Chunking / Parallel Processing is now available!

• Load - Now you can concentrate on your consumed api or connectors being of the highest quality, as time spent on one is enjoyed across your project(s). Please review Jitterbit eats REST for a potential structure to consume rest api in a standard way

An awesome benefit of designing your solutions this way is when there are bugs you have a solid hunch where it will be and with the captured payloads on all operations you can debug at design time with the data that produced the error. Priceless

Computer Says Yes! Programmer style and Reusability rocks.

Essentials for Reuse Success

Number 1 essential ingredient is a change of mindset in organisation of projects around Endpoints and Not Integration Flows Tip 4 - Project Structure

Once you organise around endpoints, reuse falls into place. You find yourself looking for reuse, planning for reuse, maintaining reuse, advocating reuse, enforcing reuse and identifying reuse.

Over time the number of objects under citizen style will outstrip the number of developer style objects. More to maintain, more to worry about.

Imagine now you are tasked to create a new integration, you can first look at the catalogue of robust endpoint operations that are already available to you per endpoint.

Naming Convention

Citizen style is descriptive but it can not be extended without getting more and more wordy, secondly it drastically reduces the effectiveness of search and impact analysis.

I choose Jitterbit's naming convention for variables standard and it really helps reinforce reuse: endpoint.object.action.name

Externalize, Externalize and Externalize Tip 10

If the same code is being used within more than one transformation, consider moving that code to a stand-alone script and calling it from within each transformation using  RunScript().   Jitterbit Success Central

RunOperationFromProject

Conclusion

Wow what a vast topic and after writing this up I realise I take for granted all the mistakes, learnings, tricks I have learnt trying to get reuse in software. I will add to the essentials over time but am impatient to publish. Please share your reuse experiences, share cool Jitterbit tricks, this is what I have learnt after 18 months and am after inspiration.