Swivel Chair integration

Any data professional should be a clear advocate of implementing good data management and governance. Like many apparently self-evident statements, this is not quite as simple as it seems.

Consider data quality. The go-to best practice approach is 'Clean at the point of capture'. Which is absolutely best practice if you have first understood why you need to capture that data, whether you have the right source or have already captured it elsewhere, what the full range of use cases are and how quality tolerances will be calculated and measured.

This is still less than half the job done though. And that's before we get into the need for a functional data governance framework to monitor, intervene and resolve data quality issues. The other problem hiding mostly in plain sight is 'swivel chair integration'. This is where data needs to be exchanged between system A and system B but no automated data integration has been built.

Hence a manual process to copy data from A to B is used instead. Sometimes via system C doing some kind of complex and undocumented manipulation. It doesn't matter what systems A and B are, but we all know system C is likely to be Excel!

I believe we need to put more focus into addressing this type of integration. Like any manual process, it's clearly going to introduce errors and those errors are going to be very hard to track and resolve. Further it is likely to be a single point of failure and it's a huge waste of valuable staff time.

It is also generally very hard to fix. Many systems were built in silo (HR, Finance, etc.) and failed to recognise the same data trapped inside these systems also lives outside. Integration tended to be built around known requirements, but never got close to addressing the myriad of use cases outside of that. Hence Swivel Chair integration became the acknowledged workaround.

Going back to automate these workarounds is often in the 'too hard' or 'not important enough' bucket. I'd strongly disagree with that. We need to start by thinking about actively managing data quality from inception to exhaust. That does not start with technology, it starts with culture. Until we can see outside of our own silos, we're doomed to confine the data to the systems which primarily support those silos.

It's an 'all court game' is you will. We cannot merely apply good data management and governance to easy to identify stations on the data journey and consider our work done.

It sets unrealistic expectations and in some cases has questionable value. For example, the benefits of this 'good' data management at that inception point are easily lost before we get to a slew of use cases for that data.

So I'm on a mission to go after these manual Swivel Chair integrations. Next time I'll talk about how to go about that.