Failure, learning and growth in digital transformation
Will Clayton
Digital Transformation Leader | Product Management | Operational Excellence | Innovation | Transforming business performance using data and technology
Most stories, articles and anecdotes of digital transformation you’ll read on sites like this will recount tales of almost unbridled success. Systems that landed on time, on budget and that far exceeded their objective benefits. In general, people would rather share their successes than defeats, and yet far more growth can come from learning from the latter.
If it is true that 70% of digital transformations fail then there must be plethora of examples out there, but where are they taking place, what happened and what can we learn from them?
I can’t write for them all, but I can certainly write for one - giving my thoughts on how despite it initially going well, what went wrong, what I learnt and how I’ve taken that experience to achieve subsequent success.
Implementing AI before it became a “thing”
Back in 2019, before the whole world was talking about AI, I was working in transformation at a fast fashion e-commerce business looking at how we could use machine learning to optimise decision making within the business. One area we selected to focus on was in Merchandising, specifically the determination of “rebuys” (aka. restocks) - second orders for products that had been launched and were selling well.
In fast fashion, getting these decisions right is worth a lot - rebuy faster and you can chase an emerging trend to increase sales, rebuy better and you can optimise stock so you don’t stock out or end up in heavy discounts.
Conceptually its a relatively straightforward problem:
Of course, nothing is ever this simple in practice, and questions like the effects of seasonality, weather and fashion trends were amongst the many we discussed between the team and our AI partner. But without diminishing the effort that went into it - feeds were established, data was ingested and models were trained. We were ready for a test.
Just do exactly what it says
Testing was always going to be a challenge. We weren’t looking for similarity with existing decisions, we wanted to see improved sales performance. As a result, we couldn’t test using a subset of historical data, as any proposed recommendations that didn’t end up as rebuys in real life (or had smaller rebuy volumes) would be missing data and skew the results.
This created a problem. Ideally we wanted to run an A/B test, but it’s not possible for a rebuy to be both placed and not placed (Schrodinger’s Rebuy) so we did something as close as possible.
For two weeks that spring, we randomly assigned every newly launched product into two cohorts. A total of 1,000 products were split in two. One cohort was to be merchandised exactly as normal and any rebuys could be placed by Merchandisers as they saw fit. The other we handed over to the AI model - instructing the Merchandisers to do exactly what it says. This wasn’t the intended usage we had in mind for future operations, but to test the models efficacy we wanted to isolate human judgement entirely.
Most initial rebuys are placed within a week or so of the product launching, so after a few weeks we had our initial data showing how the team and model had decided. But it would be much longer until the results of the rebuys themselves would become clear - we needed to wait for the stock to arrive and sales to occur.
Performing well, but failing to adopt
Watching the subsequent sales come in for a number of weeks we quickly established the AI proof of concept (PoC) was outperforming the baseline cohort.
By the end of the test period the difference was significant, the PoC had achieved an increase in sales, higher margins and an improved stockholding. To do so, the model had generally ordered faster, deeper and more consistently than the merchandisers. From a technological point of view, the PoC was a resounding success.
Yet despite these results, the solution ended up not being adopted. Following the trial, the work was passed on to the Merchandising team to refine with our development partners but after a few iterations was ultimately shelved.
But with the model performing so well in initial tests, what could go so drastically wrong to reach this conclusion?
The human side of digital transformation
Looking back, our problems had started well before we ran the PoC trials. In short, we had unwittingly fallen into the trap of focusing on the technology rather than the people. There were good reasons for this focus given the challenge it posed, but ultimately it came back to bite us.
Reflecting in detail there’s five key issues we encountered. Each one generated lessons for me and the team, and all of which have contributed to my own growth and subsequent success.
领英推荐
1. Understanding: limited AI awareness
An issue back in 2019 (and possibly even more so now given its technological growth) was a lack of understanding of AI or machine learning within the business. The pockets of knowledge were mostly within analytics or IT teams, rather than those who the technology would most impact.
As a result, as the model was being built and tested, questions would be asked like “what thresholds is it using?” or “how have you set a cut off for seasonality?”. Given the way the team operated, these were perfectly rational questions to ask, but weren’t applicable to the AI model. The model had essentially taught itself what it needed to know.
Without providing a grounding in the concepts behind machine learning, answering these questions was a challenge, leading to the next issue - trust.
2. Trust: lack of explainability
It is hard enough to provide explainability within an AI solution, let alone where the level of understanding is relatively low. But looking back at the questions being asked during the development and testing of our PoC what was underlying almost all of them was a need for trust.
The team and I were confident in the model, but we were also confident that if it didn’t work then that would be understood as being part of the process of developing a proof of concept. It is accepted that sometimes that happens.
However, for the Merchandisers their concerns were different. The model had to work if it was to help make their day-to-day decisions, and they had to trust in it whilst doing so. For them, this necessitated understanding of its decision making logic.
3. Communication: misunderstanding of intent
When we launched the test of the PoC we highlighted that the way of working it mandated was only needed for this test and in normal use the tool would be a guide, not an authority. However, it quickly became apparent that this wasn’t how it was understood.
Despite reaffirming this intent, I don’t think the message was ever clear enough - or if it was, it wasn’t believed. Combined with a lack of trust in the solution, despite (or enhanced by) its success this sowed the seeds of discontent in its adoption.
This wasn’t helped by having an unclear vision of how the tool fit into the future of merchandising, due in part to the way in which the project had been sponsored.
4. Leadership: unclear stakeholder sponsorship
The work to develop, test and establish the use of AI within the business hadn’t come from the merchandising team, but rather from Finance, IT and Transformation. As a result, to the team and its leadership it will likely have felt like being “done to” not “done with” them.
In hindsight, attempting to transform one of the core capabilities of the business as our first foray into bespoke AI, with no track record and without the buy-in of the functional leadership was probably an error. But the potential benefits from doing so were clear and I think we felt that the proof of the results would be enough to evidence its adoption - but this wasn’t the case.
5. Collaboration: competitive friction
The response to how the project came about, the impact it may have and the concerns from the team ultimately resulted in an environment of competition, not collaboration. Discussions ended up being about how one approach could do better than another, rather than how working in combination they could be the best.
This wasn’t an environment in which to succeed, with the unspoken friction it created only hindering the attempts to mitigate any concerns.
Learning, growth and success
When you’re doing something as fundamental as transforming how an entire business capability will operate, you have to expect misunderstandings, disagreements and challenges. If you don’t have some level of this, then are you pushing the art of the possible as far as you could?
Likewise, the technology underpinning such a change could be brilliant, but without people it won’t succeed. Any tension needs to be constructive and teams have to remain trusting and collaborative for this to be the case.
We didn’t quite achieve what we hoped to with rebuys, but the lessons learnt definitely helped the projects that followed it. Order App, PIM and Price Engine directly affected the same teams, significantly changed their ways of working, and succeeded in achieving their aims.
In each case, users were better involved from the outset, shared in the vision and actively supported their development - successful adoption was a natural consequence.