Independent Reviews of Official Statistics systems (2023/24) – some potential themes

1. Introduction

In 2024, Official Statistics systems remain an important and valuable part of public life in most countries. They seek to provide an independent view of the social, economic and (occasionally) the environmental state of the country and to track and communicate the various ways in which the world is changing.

There are a range of variations in the stated ambitions of different Official Statistics systems in different countries. The different outcomes that can be sought include:

• more informed, represented, empowered citizens
• more professional, accountable, transparent governments
• healthier, better-informed public debate
• more robust, successful economies

So these systems are important – but they are not static. Looking across the world, it is possible to see recent examples of significant change in what Official Statistics systems are asked to do or in the detailed roles of specific organisations. For example, in 2022, legislation was passed in New Zealand that evolved a number of key elements of that country’s Official Statistics system, including extending the formal responsibilities of Stats NZ and enabling that organisation to better access sources of data beyond surveys.

The triggers for change will differ in different circumstances but, in a number of contexts, one common mechanism for informing thinking about potentially beneficial change is for governments (or others) to commission an independent review.

The purpose of this article is to examine six independent reviews of Official Statistics systems that have been published over the last 9 months or so (between the end of 2023 and mid 2024). These reviews cover Canada, Germany, the European Union (EU), the United Kingdom (UK) and the United States of America (USA). As readers will spot, there are two reviews from the United Kingdom – a major ‘one-off’ review commissioned by Government to look at the system as a whole and the latest of a series of annual reviews produced by the regulatory body that currently operates within the UK’s Official Statistics system.

This article seeks to give what is very much a high-level take – recognising that there is an enormous amount of detail, nuance and insight in each of these reports that cannot be replicated here. The aim is simply to explore potential themes – for example in terms of the challenges that these reviews have identified or of the sorts of recommendations that they came up with.

2. The reviews

I have looked at six reports in total:

i)???The Way Forward: Addressing Challenges Facing the National Statistical System (2023 Annual Report)

Produced by the Canadian Statistics Advisory Council (CASC) on the 9th November 2023)

ii)?Recommendations of the Commission Future Statistics (KomZS)

Produced by KomZS on the 15th January 2024

iii)?ESGAB Annual Report 2023

Produced by the European Statistical Governance Advisory Board (ESGAB) on the 31st January 2024

iv) Independent Review of the UK Statistics Authority

Produced by Professor Denise Lievesley on the 12th March 2024

v)?The Nation’s Data at Risk – Meeting America’s Information Needs for the 21st Century

Produced by the American Statistical Association (ASA) in partnership with George Mason University on the 9th July 2024

vi)?The State of the Statistical System 2024

Produced by the Office for Statistics Regulation (OSR) on the 19th July 2024

?

Each of these reports summarises their terms of reference – or the broad remit of the organisations producing them. There are some clear commonalities, not least in the (sometimes implicit) assumptions about what good looks like – linking back to shared principles such as those captured in the United Nations’ Fundamental Principles of Official Statistics. There are also often commonalities of process – for example in the fact that all of the reviews draw on input and feedback from a range of relevant stakeholders and users.

There are, however, differences of emphasis across these six exercises:

• Some reviews can be characterised as ‘one-offs’ (ie there is a specific commission at a specific point in time), others have a regular nature (for example where there is a standing advisory body or committee asked to produce an annual assessment).
• There are some differences of scope – for example in the extent to which a review is focused on a single organisation, often the National Statistical Office (NSO), or on the wider Official Statistics system.
• There are also differences in the basic review model – for example whether the review centres on a comparison between current practice and some accepted desirable end-state (perhaps captured in a Code of Practice) or whether the review is asked to look at wider outcomes (for example, efficiency, efficacy, accountability).

?

3. The stated purposes of Official Statistics

There are different ways in which organisations across the world of Official Statistics express ideas about the value of their work or about the intended outcomes. Although re-imagining the core purposes of Official Statistics in any specific country is not generally central to the terms of reference for these reviews, most of the final reports do offer some thoughts about why Official Statistics matter, for example:

The CSAC report talks about the national statistical system:

• “helping Canadians keep pace with a rapidly changing society”
• “enabling Canadians to make informed decisions, lead innovation and address the country’s most pressing problems”

The KomZS report notes that statistics:

• “enable citizens to make independent and well-informed decisions”

The ESGAB report talks about statistics as:

• “yardsticks for the achievement of policy objectives, for their communication and for making comparisons between countries or over time”
• “a public good for society as a whole and for all the different user groups.”

The Lievesley report sees (official) statistics as:

• “fundamental for evidence-based policy and decision making”
• “enabling citizens to call governments to account and providing a window on society”

The ASA / GMU report covers a number of ambitions / arguments for Official Statistics, including:

• “[Official Statistics] are the official facts and figures on which countless government, personal, and business decisions depend”
• “Our democracy, economy, and society could not function without objective, accurate, timely, relevant, and credible statistics from the federal government.”

4. The challenges faced by Official Statistics systems

The reports referenced in Section 2 offer views on the challenges faced by the different Official Statistics systems in their different contexts. Recurring themes include:

Governance challenges

• The absence of sufficiently broad and deep political and social understanding of, and support for, Official Statistics
• The (linked) absence of robust and dynamic funding models for Official Statistics organisations or systems
• The specific difficulties associated with managing (or co-ordinating) more devolved or diverse Official Statistics systems (ie with multiple producer organisations)

Production challenges

• The increasing scale and complexity of the demands placed on producers of Official Statistics
• The specific need to respond to reducing survey response rates – and, more broadly, the need to ensure resilient data supply across the widening range of potential sources (including administrative data from across the public sector and wider datasets held by commercial or other non-governmental bodies)
• The methodological challenges that are now faced in seeking to maintaining high levels of quality of Official Statistics products. These challenges are often amplified by the increasing diversity of data sources used and by evolving operating constraints (for example the need to adhere to regulatory standards in terms of maintaining appropriate levels of privacy for individual level data)

Communications challenges

• The challenge in securing appropriate public profile of Official Statistics or of organisations within Official Statistics systems
• Dealing with the wider communications challenges that come with having a vast and increasing volume of information (and misinformation) in circulation

Use challenges

• The need to meet user (and potential user) expectations in terms of engagement and of voice (including user expectations of involvement and influence not just about how Official Statistics are created but also about what Official Statistics are created).

?

Challenges related to Artificial Intelligence

Finally, many reports reference the overarching challenge linked to the evolution of Artificial Intelligence (AI) technologies – often both in terms of keeping track of the capability and the reach of AI and also in understanding the direct and indirect impacts that implementation of such technologies are having, and might have, on the world of Official Statistics.

5. The recommendations

All of these reports set out a number of recommendations for action:

• The CSAC report has 4 major thematic recommendations with a number of detailed elements under each heading
• The KomZS report has 13 recommendations
• The ESGAB report has 28 recommendations
• The Lievesley report has 19 recommendations
• The ASA report has 15 recommendations
• The OSR report has 11 recommendations

In each of the reviews there is a richness of insight and detail that is relevant to each specific context. Looking across the set of reports, the following sections seek to highlight some of the recurring themes relevant to the governance, production, communications and use of Official Statistics.

Governance recommendations

It is interesting to note that many of the reviews highlight the need for further evolution of the legislation that establishes Official Statistics systems within or across countries. In some instances, the recommendations are very ambitious – KomZS, for example, call for a “fundamental reform of the BStatG” (the German Federal Statistics Act). Some of the individual legislative recommendations focus on ‘filling gaps’ – for example where current legislation might not meet what are seen as best practice standards in terms of guaranteeing professional independence or sufficiently preventing pre-release access. Other recommendations for new or amended legislation seem to be more focused on responding to more recent changes in the operating and data landscape - for example proposing enhanced powers of access to private sector datasets.

There are a number of recommendations that relate to funding. These recommendations tend to frame what is needed rather than offering more detailed models – for example CASC recommends that the relevant Government Minister should “develop new mechanisms and sources of funding to support modernization”.

Some reviews do touch on structural issues – ie about the roles of Official Statistics organisations and how they relate to each other. One common focus is on the need for more / better cross-system co-ordination – often expressing the need for the relevant NSO to have increased authority to better steer or support other producers of Official Statistics.

Some of the reviews do offer recommendations related to public sector data stewardship and to whether, or to what extent, NSOs should take on this role (see this paper for further background). Across the batch of independent reviews covered in this article, there may be some gentle differences of emphasis:

• The KomZS review recommends that, in Germany, the Federal Statistics Body is given the central Data Stewardship role for public data
• The CSAC recommendations touch on the need for Statistics Canada to be ‘a’ Data Steward within the wider Canadian public sector
• The ESGAB review recommends a level of caution about the extent to which any statistical bodies take on Data Stewardship responsibilities.

Production recommendations

Building on some of the challenges noted earlier, a number of recommendations touch on changes need to improve the quality and the reliability of data supply – whether considering solutions such as making some survey completion mandatory, or flagging actions that will enable better, more reliable administrative data access for Official Statistics producers or improve the incentives for data sharing. One interesting proposal – from the OSR Review – is for “a strategic level plan for household data across the statistical system”.

Different recommendations cover the need for investment in methodological capacity, not least at the strategic / leadership level - KomZS recommends the establishment of a Methods Advisory Board, Lievesley recommends a new senior level executive role focused on methodology.

Linked to the wider topic of funding, there is also a theme re the need for well-targeted investment in the foundational systems that will increasingly be needed for holding, analysing and sharing data – CASC notes the need for “state of the art platforms, software and communications technologies”.

Communications recommendations

Some recommendations focus on the need to increase the profile of Official Statistics. For example, the ASA review talks about the importance of increasing visibility with US legislators; the Lievesley Review proposes an annual public lecture setting out the priorities for the UK’s Official Statistics system.

There are also a number of recommendations that relate to the need to grow communications skills – for example in how Official Statistics are communicated in the ever-changing wider landscape or in the more specific techniques used in communicating uncertainty.

Recommendations related to use and users

Different reviews touch on the importance of evolving and strengthening the user engagement process in their different contexts. For example, the Lievesley Review recommends the establishment of a Triennial Statistical Assembly in the UK – to create a mechanism by which a broader range of voices can be heard in the conversation about what Official Statistics are produced and how.

Linked to that broader theme, different reviews did focus on the specific user needs of the research community, often in terms of recommending more tailored solutions. ESGAB notes the need for “easy access by properly accredited researchers to disaggregated data and microdata.”

A further theme across multiple reviews was the need for further (and perhaps more ambitious) investment in statistical / data literacy – both within government and across society more broadly.

Recommendations related to Artificial Intelligence

Most of the focus within the recommendations relates to the potential use of AI in technologies in the production of Official Statistics. OSR, for example, propose the ongoing development of appropriate standards based on or within the UK Code of Practice for Statistics.

6. Headline Conclusions

1)???? The landscape that Official Statistics operates within is highly dynamic and challenging.

2)??? Some of the challenges identified are specific to specific countries or contexts but there are recurring themes not least in terms of the challenges of increasing demand, the difficulties of ensuring resilient data supply and, of course, a range of challenges in terms of securing funding / investment.

3)??? There are some common areas of focus in terms of recommendations - for example, most reviews conclude that the legislative framework that directs and empowers Official Statistics systems / institutions in most countries needs to continue to evolve. On one issue, whether (or to what extent) Official Statistics organisations should seek or accept the role of public sector Data Steward, there may be more diversity of opinion.

4)??? Given the pace of change, it seems likely that there will be continuing value to be secured from future reviews, not least from the need for ongoing assessment of the direct and indirect impacts of the continuing growth in the capability and the deployment of AI-driven technologies.

7 A last thought

With future reviews in mind, one final reflection.

If we were looking at broadly equivalent independent reviews of other public services, my sense is that it would not be unusual for such reviews, as part of their evidence gathering, to draw on a range of quantitative information. Hence, for example, within an independent review of a public health service, it might be normal to see data on inputs (spend etc) and also on outputs (perhaps numbers of consultations or operations) and on outcomes (healthy life years). It would also not be strange to see quantitative data that gave a sense of international comparisons.

Within the various Official Statistics reviews covered here, there is in general not a lot of quantitative information. In some instances (notably in the ASA review) there is some focus on documenting different elements of expenditure but in there does seem to be little quantitative information on outputs and outcomes.

To be clear, this is not a criticism of these specific reviews, guided as they were by their specific commissions and by the information available to them. This may be more of a reflection on a systemic issue in that there is a challenge in defining and delivering meaningful quantitative data that tells us something essential about the performance and the health of Official Statistics systems.

Some NSOs do invest effort into defining Key Performance Indicators – but there may not yet be many instances where the perceived success or otherwise of Official Statistics is seen as being intrinsically tied to one or more specific metrics. More broadly, the dispersed nature of most of the Official Statistics systems being assessed through these reviews may also be a factor, given the challenges of accumulating meaningful data across multiple organisations. In a UK context, this gap in quantitative information was highlighted in the OSR report which flagged the challenge of accessing any form of data (even on inputs) on the UK Official Statistics System outwith the Office for National Statistics (the UK’s NSO).

There may be a link between this issue and the wider ongoing conversation about how best to express and demonstrate the value of Official Statistics systems. Official Statistics are often one of the mechanisms used to come to a judgement about the performance and the value of other public services. Perhaps, in order to support future reviews as well as the broader development of the sector, we may, over time, need to find better Official Statistics about Official Statistics.