How does one measure the quality of one's professional work? Call for input!

One of the books that made a real impact on me was Cathy O'Neil's 'Weapons of Math Destruction'.

Cathy O’Neil’s Weapons of Math Destruction (2016) critically examines the societal impact of algorithms and Big Data, highlighting how they often perpetuate inequality and discrimination. O’Neil introduces the concept of “Weapons of Math Destruction” (WMDs), which are algorithms characterized by opacity, scale, and damage. These systems are opaque, meaning their inner workings are hidden; scalable, amplifying their effects across large populations; and damaging, reinforcing biases and creating feedback loops that exacerbate societal inequalities.

The book explores real-world examples across fields like education, hiring, insurance, policing, and credit scoring. For instance, biased recidivism models predict criminal behavior based on flawed data, leading to harsher sentences for certain groups. Similarly, algorithms in hiring or lending often rely on historical data that embeds racial or gender biases, further marginalising disadvantaged communities.

O’Neil argues that these tools are often unregulated and difficult to contest, allowing organizations to hide behind the supposed objectivity of mathematics while making decisions that harm individuals and deepen inequality. She calls for greater transparency and accountability in algorithmic systems to prevent their misuse.

We have come a long way since 2016 as far as the legal and ethical requirements are concerned with regards to greater transparency and accountability in algorithmic systems. On example of this, is of course the EU AI Act [2025], but the principles of transparency and accountability were already coined in Article 5 of the General Data Protection Regulation [2016] as 'Principles relating to processing of personal data'.

However, human nature has, as far as I can observe, collectively, the tendency to make the same mistakes over and over again.

Back to The Weapons of Math Destruction. What Cathy describes in her book, and during lectures demonstrates with the audience, that with the best intentions, decision making often results in the opposite of the desired outcomes, even when there is no mall intent during the process of identifying the relevant indicators for the decision.

Let's look at an example from education involving the use of “value-added models” (VAMs) to evaluate teacher performance. These algorithms aimed to identify “good” teachers by measuring their students’ standardised test score improvements. Teachers whose students showed significant gains were rewarded with better pay or job security, while those with lower scores faced penalties, including termination.

So far so good.

However, what actually happened was this:

1. Opacity and Unfairness: The algorithms were opaque, making it impossible for teachers to understand how their scores were calculated or to contest them. They often penalised teachers for factors beyond their control, such as socioeconomic conditions affecting students’ performance.

2. Perverse Outcomes: Instead of improving education quality, the system created harmful incentives. For example, some teachers resorted to cheating or teaching narrowly to the test. In one case a teacher was fired not because of poor teaching but because her students’ previous teacher had inflated scores through cheating. This led to artificially low “improvements” under her tenure.

3. Systemic Damage: The algorithm disproportionately affected teachers in underprivileged schools while rewarding those in affluent areas where external factors supported higher student performance. This exacerbated inequality and demoralised educators in struggling districts.

O’Neil argues that while the VAMs were intended to improve education by identifying effective teachers, they instead punished dedicated professionals and reinforced systemic inequities, demonstrating the destructive potential of poorly designed algorithms.

So, in a somewhat different way than in quantum physics, the way we measure things, influences the thing we are measuring. And what is more: we don't measure what we set out to measure.

This introduction now leads up to the not entirely academic question I want to ask you all:

How does one determine if a data protection officer (DPO) has a baseline quality to perform DPO tasks, as can at least be expected from someone in this profession?

One way to look at this is to define a standard test in which relevant knowledge and skills are tested and in which the test identifies the lower boundary of professional quality.

But there are of course flaws in this approach, similar to the ones Cathy pointed out.

Would an additional step be to have a system of peers or mentors that provide a 'peer review' of a DPO, thus addressing the human approach, rather than trusting on test logic?

Or would other criteria be important, such as a review by the 'highest management level' of the organisation(s) at which the DPO is appointed?

Should a review from the data subjects of this organisation be taken into account?

Is the number of organisations the DPO is appointed and the total of data subjects a factor to be taken into account?

Please keep in mind that we are not trying to define the thresholds for a junior DPO, medior DPO or senior DPO, but merely the baseline quality level.

I have some personal views on these matters, which I shall love to share next weekend, but for now I want to gather your thoughts and suggestions for this generic personal question. Please tell me in the comments, or via DM what your thoughts and experiences are on this topic, and perhaps lessons learned from similar questions.