How going digital made a success of the numbers

In the years leading up to 1800, the situation in the UK was dire. Harvests were failing, bread imports were blocked due to war with France, and Parliament was worried about whether there was enough food to feed everyone. But no one really knew how many people there were to feed. Estimates based on bread production, taxes, and church records were vague at best. There were fears that the population was growing faster than food production, which could lead to famine, but without reliable data, it was impossible to know for sure.

This uncertainty led to the Census Act, which became law on December 31, 1800, the day before the UK was officially formed. From then on, everyone in the country became a number.

Censuses weren’t new, though. The Babylonians had conducted one nearly 6,000 years ago. The Romans also used censuses to keep track of citizens and their property. In fact, the word “census” comes from the Latin word “censere,” which means “to estimate.”? And, of course, the Domesday Book (1086).

For centuries, censuses around the world have been conducted on paper, requiring massive national efforts to manage, analyse, and share the data. Imagine the work involved in printing, distributing, collecting, and processing millions of paper documents. It’s no surprise that data analysis was a challenge. But technology has changed the game, and many countries have shifted from paper to digital systems. Some have found the transition easier than others.

The U.S. Census Bureau, for example, tried to lead the way with digital data collection in the year 2000, making the United States one of the first countries to use an internet-based method. However, by 2010, they had scrapped the idea due to concerns about response rates, costs, and security risks.

Fast forward to the UK's Census Transformation Programme (CTP), which recently pulled off one of the most successful transformations we've seen. Despite the challenges posed by the coronavirus pandemic, the first digital-first census in the UK achieved a 97% response rate, with 89% of households responding online. These numbers far exceeded the targets, proving just how effective the transformation was.

What made this success possible? It came down to strong leadership and a clear mission. With census day set for March 21, 2021, the leaders behind the transformation laid out a detailed plan to design a digital-first census that made sure everyone could take part. They learned from the U. S’s past challenges, addressing security concerns and ensuring that the digital systems could handle the peak loads.

Interestingly, even though this was an online census, paper still played a key role. The ONS provided paper questionnaires to those who needed them while encouraging online responses where possible. They tested their approach through the census test 2017 to understand what would work best for different segments of the population.

The level of detail that went into planning the online systems was incredible. Security was a top priority, as was ensuring the systems could handle huge volumes of activity and data, especially during the peak period on that day between 10:30am and 11:30am. Indeed, the story of our modern world will be told through data and some of the data sitting behind the census can be seen here if you are interested.: [ONS Census 2021 Digital Service]

But the real story is about the people who led this transformation to success. The leadership team’s choice to use agile delivery methods, continuous integration, and microservice architecture allowed for quick and effective changes. Decisions were carefully considered, tested, and refined.

Having joined the project in early 2017, we played a small part in this transformation and saw at first hand the journey that led to this brilliant result. In those early days, we provided expertise in programme planning, risk management and business analysis and progressed quickly to a series of high energy, proactive project managers, all with the skills and experience to deliver a multi-million pound project with many internal and external stakeholders.

Our involvement spanned through early design and development stages, working with architects, designers, developers, cloud ops, automation and cybersecurity experts at all levels up to and including deputy director. As you can imagine, there was a large focus not only on system design and development but layers and layers of data, so data architects, data engineers, data scientists all came in at different points.

Beyond the technical specialists, whole teams of people were brought in to manage the accuracy of data, so desk-based resolution teams were hired, as were many more operational people including census field support supervisors and agents, clerical matchers, HQ support staff, recruitment and campaign managers. To give you just a small picture, we alone provided over 150 specialist staff to the census and the co-running Covid Infection Survey. How many others were involved? I can't really say, but in 2021 it was recorded that the ONS employed 20,565 census field staff.

The point is that this was a huge programme and no less of a big transformation experiment. To deliver this as successfully as the ONS did, needed great leadership, a clear mission communicated effectively, teams of people who knew the part they each needed to play, deadlines that could not be missed and belief that they could deliver it on time.? For a census that was about people, this was indeed all about the people. Leaders who knew how to lead, managers who knew how to manage and staff that were selected well to do the jobs they needed to do. Round pegs in round holes.

Further, there are at least two remarkable aspects to these simple observations – firstly, that the census went ahead as planned in the middle of a pandemic and secondly, that Scotland was missing. ?

The difficulty in pulling this transformation off successfully was reflected in the decision of Scotland (and others, including Republic of Ireland) to defer until 2022.?

The ONS had belief and have been vindicated. Hats off to them.