It's probably time to retire the Hat

This is my semi-famous hat.

It was presented to me the best part of 10 years ago by my then Project Managers off the back of a rather interesting and somewhat exciting go-live for a large Australian retailer. At the time this was the largest AX Retail implementation in the world and to this day I feel privileged to have been a part of it.

For those I have worked alongside in the subsequent years you'll recall that the hat has been a feature of my desk setup and is classified for emergency use only. Those emergencies tend to be highly time (and cost) critical and more often than not data related. Thankfully there has not been a need to invoke it frequently. Below are a few of examples of when it has been required.

The aforementioned retailer, two hours before going live with the solution the highest selling product disappears. Everything, itemid, packaging quantities, onhand, etc. We cannot go live without this product so we need to bring it back to life without impacting all the other parts of the solution which have already been validated and signed off.

Hong Kong, 2015, data privacy issues. A few hours after go live it's reported that a user has noted the availability of private data and is looking to report it to authorities. Here's the issue, this potential report has the ability to land the CEO in prison. Here's the challenge, modify the security to lockdown data without impacting the ability to function and remove offending data from global tables without impacting other countries also using the solution.

This last is probably one of my favourites as there's a fairly good chance the vast majority of people involved with the solution at the time have no idea that anything even happened. It's late 2016, or thereabouts. One of the users has managed to accidentally delete the entire batch schedule from the system. Being an Enterprise customer heavily reliant upon batching this is going to be a little bit of an issue. Here we see the advantage of being the Support Manager at the time, decide to seek forgiveness and not permission. Put the hat on, issue resolved in sub-20 minutes with many none the wiser that anything had even happened.

The hat could also be called 'The Hat of Plausible Deniability'. As there are rules in place when it's use is invoked. One of those rules is you really don't want to know how the fix is being done. Remember emergency use only and when you're in a situation where each hour of downtime can cost the business upwards of $150K. Time is critical and there is not really a lot of time available for change by committee. But before I get the Process gurus jumping down my throat, this article is not about process or lack thereof.

You see, I'd like to retire the hat but alas it seems that as an industry we are still falling into the same traps we were falling into way back then.

The general challenge here is Data Migration, a crucial part of any project. Often overlooked to a degree but critical to the initial operations of any new system. I'm not referring to the 'easy' stuff, all my customers are in place, counted the rows. Yes the Trial Balance matches. There are templates for that and they're relatively simple to use. While those items are essential to a successful deployment a missing customer record or an unreconciled trial balance can be dealt with after the event if necessary.

What I'm referring to are the inflight transactions, the Received Purchase Orders, the Dispatched Sales Orders, the reconciliation of the stock on-hand but not only to the physical but also the reservations and on order values as well. If you look at it, how do you dispatch an item if it isn't available in stock? Or make a reservation for something which isn't on order? Or how about this one. Keeping in mind I've seen this happen. You run your first Master Plan only to find it attempts to re-purchase everything which has been partially delivered because the status of the underlying Sales Lines could not be determined. These types of issues are not known to instill confidence with the client regarding ongoing operations.

I believe part of the issue is around how the Data Migration is dealt with within the Project. Below are three common approaches I've seen to addressing these challenges.

Vendor Driven - This is where the Implementation Partner will also take on the migration. Generally a technical exercise and may not necessarily be a specialty of the resource\s being utilised. Why? Data Migration is a highly specialised skill and generally a vendor will not have that capability sitting on the bench awaiting the next rollout. Expectation here is that they know the end points and as a general rule that is true, often done using the available templates the same way as everyone else.

Client Driven - The Vendor provides the templates and the Client fills them. After all it's our data, we've been using it for years and have a group of people who know our data intimately. Yes all very true but what they do not know are the end points and they are yet to understand the intricacies of the data interactions within the new solution. Generally this team will be made up of current client side report writers or database experts. It's almost the direct opposite of the Vendor Driven approach where they know the From state intimately but have limited awareness of the required To state.

External Team - This is where you go to market to build a team specifically for the purpose of doing the Migration. It'll tend to be made up of contractors employed for either the duration of the project or partial according to the implementation plan. Now here's the thing, this team will need access to both the above mentioned Client and specifically in the case of an ISV solution implementation the Vendor team. In the current climate it can also be quite a challenge to build this team given it would generally be made up of Implementation resources who are currently between implementations. With so many implementations and upgrades underway at the moment, resources with migration experience are extremely difficult to find and as already mentioned it's a fairly specialised skill.

So here I am once again being the Harbinger of Doom. There's not a great deal in the above which will allow you to nail the data migration while allowing me to retire the hat.

That said I was recently exposed to a tool set which deals with the above mentioned concerns. You see the thing is that Data Migration is actually a repeatable process but tends not to be looked upon as such at the project level. While it's true that often tools may be built as part of the project they are generally specific and bespoke to the project itself. What I've recently seen is a tool where the specifics are the Data Migration. In this instance I'm referring to migration to D365 Finance and Operations. I'm assured it goes beyond that although that's the area I'm really interested in.

This toolset will highlight your validation and any relational mismatches. It deals with the transformation elements as well. It's created to perform the job of the potential teams mentioned above and is executed by a team who specialise in Data Migrations. That is all they do and they do it well.

You can also automate your test builds. Think about it, you can actually test your solution with migrated data. Not a subset, not well if you enter a few customers in you should be good to go. But an actual example of that true data which will be migrated and you can do it multiple times, with ease.

So just maybe it will be possible to retire the hat after all. I'll miss it as we had some fun times together. Although I know a number of Deployment Managers and other Process orientated folk who may sleep a little easier this evening. Thinking about it though this may only be a form of semi-retirement as the next Priority 1 incident could be just around the corner, you just never know.