Can we use AI for public sector board and committee minutes?

Outcomes from some early experiments

Everyone’s talking about AI, and all its potential uses. And plenty of people are already recommending it for minute taking as if it’s a “no-brainer"[1] [2].

But how good is AI for summarising formal public sector board and committee meetings?

Minuting such meetings is highly skilled work, requiring subtle analysis, understanding and interpretation from the minute-taker. The art of committee minute-taking is also governed by widely accepted stylistic rules and other conventions.

AI is amazing – but can it really replicate that very sophisticated human skill, which is sometimes wryly described as, “Recording what the Board will think it ought to have said”?

I've recently been experimenting with the Microsoft Teams AI-powered Intelligent Recap function for committee meetings. This article summarises some personal observations to date, from comparing my own perception of what was said in the meetings with what the AI summary captured.

I've also highlighted some governance risks that AI minute taking could create.

Spoiler alert: I don’t think this app is going to threaten any good committee minute-taker’s job as things stand, although who knows what the future holds.

This article comes with all kinds of caveats.

Above all, just a reminder that Pushing the Pen is completely separate to my day job, and does not necessarily represent my employer or its views.

Other caveats include the fact that I know very little about AI, and am on a steep learning curve. So there may well be some ignorant generalisations or omissions here. This whole article is based on just a few weeks of using the app and my own subjective views; so it may well be unfair or even inaccurate in places.

Also: it’s worth noting that MS Teams Intelligent Recap has been rated by some as “low” on its ability to summarise key points from meetings, while other apps such as Otter.ai and Fireflies.ai apparently do much better [3]. However, I’m writing about Teams for the simple reason that I have internal Information Governance (IG) clearance to use it.

Whether we like it or not, Microsoft applications seem to be dominating the public sector, so that’s realistically what we need to work with at the moment. It may well be that there are better apps out there for formal minute-taking; but if you’re not allowed to use them for your own work that's probably a dead end.

Therefore, in the public sector context, there's seemingly little mileage in comparing and contrasting different apps for AI minute taking at this point in time, although this could of course change. Without IG clearance, there are risks of feeding confidential corporate information into a database without understanding how that information will be stored, used and kept secure.

With those caveats, here’s a summary of some early observations on the capabilities of MS Teams Intelligent Recap for board and committee minutes . . .

What looks good?

Overall summary

The spelling and grammar look very good. The MS Teams transcripts can be prone to swathes of misheard “nonsense”, depending on how clearly people speak. The AI recaps don’t seem to have the same problem, which in interesting as they are apparently generated from the transcript. I've only picked up very occasional misuse of homonyms, for example reference to a “draught plan”.

It seems to do well on disregarding “actions” which aren't true meeting actions. For example, the chair asking the meeting secretary to perform routine / basic meeting admin tasks in real time.

There are also some very impressive touches, albeit these may not be consistent enough to be relied on. For example, in one meeting an action was agreed and then later added to – this was picked up correctly.

The AI also occasionally does a clever job of grouping together several diverse but connected points, although the finished summary for complex discussions like this is generally unintelligible.

Intelligent Recap seems to perform best on things like routine team meetings, or catch-ups with a small number of people. For these kind of meetings, it can be brilliant at summarising a discussion succinctly and well.

This may indicate that the app works better on smaller and more operational meetings, although obviously I wouldn't draw any firm conclusions yet from such a small sample and short time-frame for testing.

Other functions of Intelligent Recap

It’s worth noting that Intelligent Recap does more than just take meeting notes. It also generates timeline markers which show when people speak, or join or leave the meeting; or when a screen was shared and so on.

It automatically generates meeting “chapters” to help with understanding the meeting, and navigating the recording. It also offers live translations (for captions) into multiple languages.

The wider Teams Premium package includes various other functionalities such as Advanced Virtual Appointments and “Town Halls”. This article doesn’t have scope to describe all the features of Teams Premium / Intelligent Recap in detail, but you can read about it on the MS website [4].

What’s not so promising?

Stylistic conventions

The summary isn't a set of formal meeting minutes as we’d recognise it. The software doesn’t seem to recognise things like agreeing the previous minutes, declarations of interest, establishing that the meeting is quorate, or noting decisions at the end of an item. So the minute-taker would need to ensure that all the necessary information was captured.

There doesn't appear to be any way to generate minutes already formatted within your own corporate template, although this could well be available in the future.

Verb tenses

The summary is mostly written in the simple past tense which is an appropriate base for meeting minutes. It occasionally uses the future perfect tense correctly ("XXX expressed that XXX should be included in the project"; "XXX suggested that XXX could consult other colleagues on the questions").

However I've not yet seen it use the past perfect tense ("The Chair had already spoken to XXX"), which an expert human minute taker would instinctively use. In such cases it seems to prefer the simple past tense ("The Chair spoke to XXX"), or present perfect ("XXX has proactively addressed the issue").

This is because the AI is directly transcribing what the speaker said in the moment. To meet the accepted conventions of formal minutes, someone would therefore need to edit the text in this regard.

Recording actions

The accuracy of recording complex actions is often poor. For example, Intelligent Recap often seems to generate actions when people talk about something that they're already working on. So if someone says that they're drafting an XXX plan, an action is sometime noted for that person to draft an XXX plan.

It also tends to simplify actions, sometimes rendering them unintelligible in the process. Other times, this simplification changes the nature of the action. For example, one meeting action was agreed with a caveat, but the caveat wasn’t recorded.

Another weakness is that the recap sometimes incorrectly records actions where a possible course of action has been briefly discussed but not agreed.

Meeting summaries in general

The summary of complex discussions can be very muddled. Sometimes two or more topics are merged into one point, which makes for a confusing summary.

Equally, the recap often just describes the topic under discussion at a high level – “XXX and XXX discussed XXX”, while failing to capture any key opinions expressed, or questions asked.

It also sometimes records trivialities as if they were part of the meeting:

-?????? XXX asked if they should stop recording the meeting.

-?????? XXX asked if XXX was still there.

Another weakness is the tendency to repeat the same point in the notes – sometimes a sizeable chunk of text can recur as often as three or four times, almost word-for-word the same.

Voice recognition

If a meeting is held remotely, Teams can easily pick up who is speaking, as everyone’s computer has its own microphone. However, the software is currently unable to differentiate between several speakers sharing a device in the same room (eg, a single laptop or the room video-conferencing).

This would make the AI recap function harder to use for face-to-face or hybrid meetings.

However, some options appear to be emerging to make it possible to differentiate speakers:

• Intelligent speakers for Microsoft Teams Rooms – smart devices in a room that can differentiate voices [5].
• Enrollment of voice and face for Recognition – training your Teams account with your voice and face ‘profile’ so that can then be used when in a hybrid setting [6].

Hallucinations

A really concerning problem has been the appearance of “hallucinations” in the AI notes. Where things haven’t been clear to the software in meetings (e.g. the name of the agenda item wasn’t explicitly spoken aloud), the AI has proactively generated section headings, which have varied in their appropriateness.

It’s also gone further on occasions, and “invented” things – for example in one set of meeting notes it referred to a national strategy which doesn’t actually exist. Scarily, this fictional strategy did sound plausible in the context of the discussion. I found it uncanny to see words which had not been spoken at all in the meeting, but which were perfectly appropriate and natural (although wrong) for the context.

Apparently this is a well-known issues called “AI hallucinations”. Zapier writes [7]:

An AI hallucination is when an AI model generates incorrect information but presents it as if it were a fact. Why would it do that? AI tools like ChatGPT are trained to predict strings of words that best match your query. They lack the reasoning, however, to apply logic or consider any factual inconsistencies they're spitting out.

In other words, AI hallucinations occur when the AI goes off the rails trying to please you.

It’s worrying to think that an AI summary of a formal public sector meeting could include false information. There could be a risk of people not noticing this, if they trust the summary too much. Public sector meeting minutes are very important documents which are routinely used for audits, inquiries and legal matters, or to give assurance to the public and other stakeholders. They also have a core function to accurately codify corporate memory.

Other risks of using AI for public sector committee minutes

As well as the threat of hallucinations, and the information security risks mentioned above using AI for public sector board and committee minutes presents several other potential governance risks.

One significant risk is that organisations and individuals could overestimate the ability of AI to produce adequate formal meeting minutes. This could lead to consequences such as deleting jobs in the belief that AI can do administrators' jobs "better", or the production of unsafe, poor-quality minutes because no one is checking them properly against what was actually said.

There could also be a theoretical risk of AI programmers with strong ideological beliefs encoding some kind of bias into the algorithm, so that the recording of discussions and even decisions are subtly skewed in some way. This would be untenable for the public sector.

Intelligent Recap also appears unable to pick up key nuances or context that a skilled human minute-taker would naturally have regard to in their drafting.

An excellent article from Torys Quarterly [8] identifies several further governance risks, as well as their possible mitigations. These risks include:

Litigation risk

The creation of robust and detailed—and unvetted and unfiltered—meeting minutes generated by AI-notetakers may lead to an increase in the availability of information that may be discoverable by third parties in litigation.

Privileged communications

If AI-notetaker settings permit meeting minutes or summaries to be distributed automatically to attendees, legal advice intended to be privileged could be circulated without appropriate vetting, notification or consent.

Chill on debate and oversight

Employees and directors [may] be more reluctant to engage in healthy debate […] where AI-notetakers are preparing the minutes, with less human review and discretion over the final content.

Bias and impact on culture

AI-notetakers can sometimes exhibit bias by giving deference to senior people at a meeting.

However, there are also some promising opportunities, if we can find a safe and effective way to harness the speed and power of AI to make people’s working lives easier.

Conclusion

The experimentation so far appears underwhelming in the context of public sector board and committee minutes. For complex meetings, my own (subjective and limited) view is that it hasn’t managed to add value for the minute taker yet, or produce anything better than (or anywhere near as good as) a human secretary can.

No matter how impressive it seems, Intelligent Recap is still just a meeting support tool. It isn’t currently able to process complex board and committee meetings into professional minutes, and a human secretary and senior sign-off will still be essential. However, there's a lot of work ahead to understand what it can and can’t do, as the technology evolves over time.

Elements of the meeting summaries are astonishingly good, and this surely indicates some great future potential for this technology. However, as things stand, this brilliance seems to be too variable and unpredictable to rely on.

It's possible that the role of the meeting secretary will change, from someone who summarises the discussion, to someone who edits an AI summary of the discussion. At present however, it seems to me that this form of editing would be no easier, faster or less skilled than writing the summary yourself.

As always I can’t (and have no wish to) speak for my organisation or sector, but my view is that AI is here to stay, and will continue growing stronger. It therefore feels more sensible to seek a clear grip over what it can and can’t do – and deeply understand any risks at stake for governance – than to passively wait for the AI to "take over" or find ourselves pushed into using something that we don't really understand.

?* Disclaimer: Pushing the Pen is completely separate to my day job, and does not necessarily represent my employer or its views.

You are welcome to use share or adapt this material for free under Creative Commons: CC-BY-SA 4.0, as long as you credit Pushing the Pen as appropriate: https://creativecommons.org/licenses/by-sa/4.0

Image credit: A Woman Holding a Digital Tablet | Michelangelo Buonarroti via Pexels

[1] Hate Being the Meeting Minute-Taker? Fire Yourself and Hire AI. | James Rowe via LinkedIn

[2] Using AI Tools to Make Meeting Minutes Magic! | The PMI Newsletter via LinkedIn

[3] Meeting Minutes with AI: Transforming Productivity in the Digital Age | JJW Project Solutions

[4]: Microsoft Teams Premium—the smart place to work | Product information page from Microsoft

[5] Announcing general availability for Intelligent speakers for Microsoft Teams Rooms | Microsoft Community Hub

[6] Overview of voice and face enrollment | Learn Microsoft

[7] What are AI hallucinations and how do you prevent them? | Zapier

[8] AI-minute taking tensions: navigating risks and best practices | Torys Quarterly