What are “Microdata?”

Microdata are individual records that represent a specific person, household, establishment, or another unit of analysis. The photo above shows microdata in their most visceral form: demographic information that represents a person being punched into a piece of cardboard, part of the automated data processing for the 1940 US Census.

If you create a survey with a system like Google Forms or Qualtrix, these services will let you download the survey responses as a single spreadsheet. Here, the microdata are the individual rows of the spreadsheet—each row is an individual survey response. These are the microdata in their most raw and most difficult-to-consume form. That's because these data can be incomplete (not every question gets answered) and incorrect (not every question may be answered correctly).

When I joined the US Census Bureau back in 2017, one of my biggest surprises was learning that the Census Bureau edits the survey responses that it receives. I had always thought that administering a survey was like measuring the pH of an acid in a chemistry lab: changing the data that you collected was somehow akin to scientific fraud.

I have since learned that it is common to edit survey responses. Sometimes the editing is as simple as throwing out the microdata that are incomplete or obviously incorrect. But editing can be far more involved. What's important is to make sure that all of the data are edited using rules that are both clear and that are consistently applied. Otherwise, the editing can do significant harm to the scientific validity of any analysis for which you might use the data. (Dan Bouk's book Democracy's Data: The Hidden Stories in the US Census and How to Read Them, goes into detail about some of the edits that were made as part of the 1940 Census.)

Students in some data science courses frequently download and analyze publicly available microdata. For example, the Integrated Public Use Microdata Series USA website, better known as IPUMS USA, allows students, researchers, and pretty much anyone with a valid purpose to download a range of microdata from the US decennial census as well as the more detailed American Community Surveys (ACS). These datasets contain individual records that represent individual people, bound together with an additional record that represents the residence in which they lived—and in some cases, may still live. That is, these are real data from real people. For this reason, the IPUMS website extolls researchers to "USE IT FOR GOOD—NEVER FOR EVIL."

(The creator of IPUMS, Steven Ruggles, was recently awarded a MacArthur Foundation fellowship in recognition of his work "setting new standards in quantitative historical research by building the world's largest publicly available database of population statistics.")

The data collected by the US Census Bureau were collected under a pledge of confidentiality, so the microdata released by the Census Bureau are processed before they are released to make it difficult for researchers to identify the specific person on which each microdata record is based. For example, instead of reporting each person's street address and city, each record is instead located within a Public Use Microdata Area (PUMA), a statistical geographic area that contains no fewer than 100,000 people.

Here are some sources to consider if you are looking for microdata:

• IPUMS, with more than nine different broad categories including demographics, global health, geography, time use, and higher education.
• The World Bank Microdata Library, with over 4000 datasets.
• The US Census Bureau's Restricted-Use Microdata, which can only be accessed from the Federal Statistical Research Data Centers by qualified researchers who have undergone a background check and have a pre-approved scientifically valid research project.

Statistical microdata are useful for academics conducting research, but they can also be useful for businesses trying to understand potential markets.

_____________

Simson Garfinkel was the Senior Computer Scientist for Confidentiality and Data Access during the 2020 Census and was then the Senior Data Scientist at the US Department of Homeland Security, where he worked on data governance issues. His most recent book Law and Policy for the Quantum Age is now available as an audiobook from Audible.