How to make an audit go wrong

There has been a major scandal in the UK recently, around the Horizon software for the Post Office. At the heart of the scandal is bugs (errors) in the Horizon software used to manage financial transactions.

Sub post masters are the people who manage mostly smaller local post offices. They are responsible for making sure that the contents of the till match up to the amounts recorded. This means the Horizon software also doubles up as a way to audit them. The affected sub post masters were accused of fraud, many were fined or lost their jobs, a few went to prison.

I've been waiting for a technical explanation of the root cause, but it hasn't appeared in the news yet.

Interestingly, the problem affected some sub post masters regularly, but others not at all. There's a hint that it might be to do with the work pattern / practices of some sub post masters. I've worked in support teams fixing bugs, and this is unusual. Most bugs affect everyone.

The other interesting feature is that the bug never seemed to work the other way, and show the sub post master had spare money. Only when it was not enough. It's possible that the investigation is only reporting on the not enough case, but otherwise, this seems odd.

I was wondering about what kind of scenario might have these characteristics.

Disclaimer

I'm not related to any part of the Horizon software team, the post office, the legal cases or the inquiry. I have never seen any information outside public news reports, and I have no idea whether this is actually what happened. Or whether this has happened at any company. I'm just speculating.

Possible scenario

1. Over the course of a month, make payments, add and remove money.
2. On the last day of the month, in the morning, run Audit A
3. On the last day of the month, in the afternoon, add or remove a bit more money
4. On the first day of the following month, run Audit B

You now have two audits, A and B, showing different numbers: apparent discrepancy.

There are of course many ways to avoid this issue, a few suggestions:

• Lock down the system once audit A has been run, don't allow any more edits or entries into that month
• Mark entries as audited once they have been audited once, optionally with a timestamp. If the audit runs again, warn the user that some entries were audited before, and others not
• Save audit A into a different database table, and reload it on request, instead of re-running the audit query

Incidentally, if you have ever worked on a system that locked down for a few days around month end or year end, now you know why!

One observation about this theory is: there's nothing wrong with the code. Automated tests won't find issues unless someone thinks about the second audit scenario. It's an architectural bug, not a code bug.

What do you think? Does this sound possible? What do you think might be the root cause? (Please don't comment if you are in any way related to the investigation, the inquiry is still under way).

#debug #root cause #coding