Local Councils, Why you're not ready for AI! and what you can do about it

Over the past few weeks I attended a few public sector data events and as you can probably guess, the topic of the day in all of these events was AI and everyone was flocking to get a piece of it.

I had conversations with countless IT leaders, engineers, business managers and more, and attended several seminars. Everyone was there to find out what and where they can use AI for, what products they should look into, how they should implement it and so on.

At the start of the day you'd think we're on the cusp of the artificial singularity and at our next general election we'd have an artificial super intelligence running as an independent candidate for prime minister, but alas by the end of the day the sun sets on the wonderful aspirations and dreams, and reality dawns, that the majority of local councils are no where near ready for any form intelligent analytics let alone "AI".

Now, just before I get into the reason why I think this is the case, I want to be clear. This is not a punch down on these incredible organisations that maintain, manage and safeguard our civil infrastructure and way of modern life. Rather, I'm merely echoing the pain that many data professionals and subject experts within these organisations have shared with me. So if you disagree with these points, write a comment and tell me why, maybe I have it wrong or maybe what I'm describing are isolated cases, if you do agree, then make sure you let me know and share it with your colleagues!

The Data culture

Data, people and infrastructure supporting it, continues to be an ancillary item, rather than a strategic one.

Think the IT Crowd type scenario. Key executives access and consume reports, yet aren't really aware of how these reports are formed, where the data comes from, how its cleansed, verified and how it is brought together. They only see the end result, a pretty spreadsheet or a one slide on a power point with a summary paragraph and nice little pie chart.

The heroes that power the data intelligence infrastructure in many local government organisation put in an immense amount of effort, time and energy to make it work with the tools available to them.

The age old claim of data professionals spending 80% of their time wrangling data and less than 20% of it analysing it, although may lack the evidence backing it in 2024, does seem to ring true when asking a data professional in the public sector what their day consists of.

At the very least they'll spend 80% of the time answering that question explaining how they so often with their "Bare hands", manually scrape information from various sources and correct errors through thousands of rows across dozens of spreadsheet and then having to email their output to the next person in the process to do their bit, in what can be only described as a data conga line.

Some of the data professionals I spoke with exhaled a sense of frustration that there is a level of detachment from the data and the opportunities they can uncover, from senior leaders. They see the data as an admin task to report on "the real work they do".

There is also the fragmented approach and lack of unison across organisations with regards to data strategy. I spoke with a few department heads that explained that they have to use their own initiative to define process and policy within their units. With respect to their abilities and approach, we can certainly describe them as data mature parts of the organisation. However they rightly conceded that their progress, which left many departments behind, unintentionally created a culture of silos. A culture where a department owns the data that is specific to them, uses their own tools, process and formats, as oppose to it being owned by the organisations under a unified approach of process and policy of governance.

The domino effect continues as now access to data from across units, becomes a challenging task. An exercise that should take no more than a few hours, at best takes several days, or more often, several weeks. Concern's pertaining to data access and sharing, as well as unit priorities underline the rigid bottleneck that is the cross-departmental and organisational data sharing mechanism.

Which leads me to the common data estate across these organisations.

Cumbersome, Fragmented, Siloed, Untrusted, Ungoverned.

I should have created a word cloud for the most common words I've heard throughout the events, but you can just imagine I did. In essence these five words would be the big ones that would stick out. Every local government organisation seems to have this identical chronic debilitating challenge of inaccessible, untrusted, bottled neck data, where information is sitting in a number of systems managed by people they're unaware of, information lacks a level of certainty about it, and requesting access to it can be as complicated as trying to describe a colour that doesn't exist.

The varying approaches of whether departments use off the shelf tools, or deploy their exuberant skills in scripting and hand cranking the machine, often yield desired results and positive outcomes in the short run. However as one statistician mentioned to me, in the long run those gains are eroded by the complexity of the estate and the lack of documentation snowball maintenance costs into an avalanche that lead to projects being canned, or forced to start over again.

Note, I haven’t even touched on the topics of governance or the lack of any policy pertaining to how generative AI models can be used in the organisation. That’s because the former is a static set of rules that debilitates innovation rather than facilitating it safely, and the latter simply does not exist... yet.

Therefore its fair to say it is not appropriate to deploy machine learning or AI models when data is inaccessible and untrusted and above all, data is ungoverned.

The upside, is the aspirations. All of the organisations I engaged with possess a vast amount of data, accompanied by impressive use cases- from predictive analytics and geospatial heat maps, to public data analysis and cross-organisational data sharing. The potential of this information to create substantial impacts and enhance citizens' quality of life is immense. I can happily exclaim that no private organisation could match or claim the transformative influence that a local council can have on its community. But this claim is only true if our local councils and other local public organisations are willing to fully commit to their digital journey. No half measures! one bad apple can ruin the entire box.

So what now?

Culture

Data within local public organisations can only be described as the cyber twin of the reality that they govern, the assets they hold and most importantly the people they look after. Therefore in my humble opinion the best data culture is the one that drives forward an organisational wide consensus around the importance of accessible, trusted and governed data, and the goals they will serve.

In doing so, all members of the organisation would naturally work towards upholding these principles, recommending best practise and encouraging innovation. For example, identifying when there is a lack of policy regarding the use of genAI. In return the culture will allow you to chart a path towards identifying opportunities, negating risks, improving services, encouraging collaboration and managing resources effectively.

Data strategy

I'll keep this one short as I've touched upon this topic in a previous post Crafting an effective data strategy. However what I would add here is this:

Your Data strategy should not be a footnote in the annual report, but rather intertwined with the general direction and strategy of the organisation as a whole.

It's 2024, technology practically underlines our entire existence and it will only get more technological as our future becomes our present. One can no longer afford to create separation between the ones and zeros and the brick and mortar. Your data strategy is your strategy.

The Tech Eco System

This will need an article to itself, but I'll keep it brief.

Don't reinvent the wheel. Invest in your infrastructure, and acquire the off the shelf solutions and then combine it with your teams exquisite scripting skills.

Simply put, the capabilities you need, to break down the silos, to move, transform and improve the quality of data, to store, visualise and analyse the prepared information, to govern it and manage access controls are all products available in abundance right now. They'll allow you to standardise, automate and federate your data management and analytics activities, in a governed manner. What isn't in abundance is your time or the skills needed. Which is why it is important to focus on key priorities and that is delivering the insights your organisations needs. And it is not a bad thing to get external support in these endeavours . Software vendors and their partners can come in and support your technology selection, implementation, delivery and enablement as well as ongoing support, so you can focus on analysing and making data driven decisions.

So if by the end of this you're wondering can my organisation deploy AI?

The answer would be obviously yes! you can do anything you want. But really what you should be asking is, Should my organisation deploy AI? and the answer will be tied to how you answer these questions

Can you trust your data? Can you trace the lineage of data?

Is your data easily accessible? Can you share it?

Is it governed? Do you have policies regulating the use of machine intelligence?

If it's a yes to all of these, you're off to a good start, and can now indulge your appetite for risk surrounding the use of it.

If not, then you are a distance away from fully realising machine intelligence capabilities in your sphere of work and you should start working on improving your data assets and how you manage them.

AI in all its forms is a fascinating topic that has a remarkable potential for good and bad. Make sure you're clear about what you want to use it for. Relish the positive outcomes it'll provide, but keep in mind the words of Dr Scott Zoldi:

Machine learning and AI can become callous. Particularly if it’s starting to provide value. Very often people don’t inspect where it could go wrong. Because when AI is wrong, it is very wrong!

Thanks for indulging my take on this topic of AI readiness in the public sector. Feel free to share or more importantly let me know your view on this. agree or disagree, I'm always keen to learn and hear a different opinion.