Data Program Disasters: Unveiling the Common Pitfalls

When I propose the idea of creating a Data Platform, what images immediately flash through your mind? Let me attempt to subtitle your thoughts. Perhaps you envision a dazzling array of cloud providers, data transformation marvels, powerful governance tools, and programming languages like Python or Scala. Snowflake and Databricks might be among the shiny tools at the forefront of your imagination. These cutting-edge technologies have captivated not only the techies but even the executives saddled in the comfort of the mahogany desks in their ivory towers. So it's natural that more than 99% of budget allocation happens to these polished packages. But what about the fundamental pillars of success: Culture, Communication and Change management, in a word, the People? Sadly, we take them for granted, allocating less than 1% of the budget to these crucial aspects. Isn't embarking on a critical mission armed only with a butter knife to cut through the thickest of thickets?

Respect your in-house Talent: Data Platform gleans data from the existing systems and curates and makes it accessible for the betterment of the consumers of the system. So all along program is "for the system and by the system" Then why do organizations jump at the very first instance to recruit expensive external recruits to design and execute such programs? Understandably tech stack may vary, and we will need fresh talents, but it should not become an excuse to undermine existing talents. Countless data programs have faltered because management becomes overly fixated on assembling a completely new team, expecting them to deliver revolutionary improvements even before familiarising themselves with the system's intricacies. Respect the in-house talent and their tribal knowledge; it's a treasure house. Invest in upskilling them and empower them to drive these initiatives. Look at industry leaders like Shell, Johnson and Johnson, and Airbus; they represent a minuscule fraction of the world's successful organizations that have implemented data programs and are now reaping substantial benefits. While they vary in business model and domain, the common thread that bonds them is their emphasis on the "Citizen Data" program, which places their employees at the forefront of this radical transformation. While external experts and specialized vendors certainly have their place, relying solely on them to guide you towards success is like placing a blindfolded artist with a paintbrush in front of a canvas and expecting a masterpiece.

The Role Dilemma: Unlike application development, Data is an emerging field, and its roles are evolving. There are glittering roles with overlapping responsibilities, such as Data Analysts, Data Engineers, Data Scientists, Data architects, stewards, modellers, etc. However, it's crucial to avoid being swayed by the allure of these terms, as they can sometimes be used to inflate rates and, thereby, the monthly bills. Does your project need and understand all these roles? Possibly not. Are your managers leveraging the right person for the right roles? Let's conduct a revealing acid test. Reach out to your leads and ask them to explain the intricacies of each data role eloquently. Prepare to be astonished as you will discover that not even three of your leads will converge on a shared definition, I'm willing to wager my paycheck on it. This indicates that teams are devouring your budget without significant contributions to progress. It's vital to return to the fundamentals and truly comprehend your needs. Though it may seem like slowing down the project, remember that remaining grounded during inclement weather is not a sign of weakness.

Data Quality stands as the formidable adversary capable of undermining your program's success. Devoting careful consideration to delineating the boundaries of "Good Data" becomes imperative. While attaining absolute purity in data may elude us, adopting disciplined methodologies for cleansing and curating empowers us to reach a commendable level of quality. This necessitates enlisting the expertise of subject matter experts to address prevailing gaps and zealously mend the orifices through which flawed Data infiltrates our system. Any tardiness in this segment will render your data system completely useless, just like a grand ship, meticulously crafted and adorned, yet destined to gather dust in a forgotten harbour forever.

Wonder ToolKits: In the realm of data management, it's tempting to be swayed by the enticing promises of new tools that offer lineage, provenance, cataloguing, observability, and more. However, beneath the glossy marketing exterior lies the lurking devil of hidden costs that can burn a hole in your wallet. Let's consider an example: while you may have successfully negotiated a reduction in compute costs, you might have overlooked the expenses associated with data egress. This oversight could lead to long-term vendor lock-in or force you to spend the hard-earned savings secured through skilful negotiation on the data outflow. This is just one instance among many; there are live examples where organizations have chosen tools solely based on their features and figured lately that such tools needed to fully comply with the industry's regulations or the country they operate in. In such cases, you're left with two options: either wait for the vendor to become compliant, severely stifling your Go-To-Market strategy or supplement your setup with additional services, effectively negating your cost-saving efforts and bloating your architecture. Evaluating tools in this complex landscape is a precarious endeavour, as it's easy to be blindsided by one or more critical aspects. Falling for the marketing spiel is like admiring a beautifully wrapped gift without ever opening it...

Data Programs may seem daunting, but it's not their complexity that trips us but our misguided priorities and the intoxicating allure of shortcuts that make them overwhelming. To unlock true success, we must reconnect with the fundamentals, respect, recognize, and reskill our in-house teams, avoid the temptation of glamorous roles, relentlessly focus on cleaning our data, and scrutinize every detail of the new toolkit. Remember, these steps offer a minimum guarantee of triumph; the rest hinges on our unwavering commitment to execution. So, the next time failure taints and taunts a data program's name, remember: it's not the program that faltered but the approach that betrayed us. In conclusion, remember, it's not rocket science but just common sense!