Tales from the Data Darkside

Data science has many benefits for citizens. Sound data engineering has helped governments create and deliver better government services. Democracy thrives on good data.

Data can also be misused, often leading to horrifying results.

I recently read The Hank Show, which details how an entrepreneurial house painter turned drug runner turned DEA informant built a business empire based on buying states’ division of motor vehicles (DMV) data. Hank Asher, the “Father of Data Fusion,” pioneered many methods to combine databases to provide in-depth profiles of people. He started by buying Florida DMV records and cross-referencing people with buying records, legal records, and other government data sources to track peoples’ data trails.

Hank Asher would sell his data products by going to law enforcement trade shows and asking visitors for their names and addresses. He would plug the information into his data systems and display personal information about the person. The book recounts one story where a sheriff asked for a demonstration. Up popped the sheriff’s personal details, including the fact that he had been previously married – which surprised his current wife.

What’s the dark side of Asher’s inventions? As the author, McKenzie Funk, writes:

“A world in which computers accurately collect and remember and thereby make decisions based on every little thing you’ve ever done is a world in which your past is ever more determinant of your future. It’s a world tailored to who you’ve been and not who you plan to be, one that perpetuates the lopsided structures we have, not those we want. . .. See yourself through Asher’s inventions, and you see that what may be scariest is not when the machines are wrong about you—but when they’re right.” (p. 12)

And when the databases are wrong, they can have disastrous effects. There are many stories of people wrongly accused of crimes because of bad data. Other people have been denied the right to vote because of misleading data. Then, there are the recent stories of how COVID-19 contact tracing data was misused, causing privacy concerns.

Rob Copeland’s The Fund: Ray Dalio, Bridgewater Associates, and the Unraveling of a Wall Street Legend was the second book offering a cautionary data tale. I first heard about Bridgewater’s unique culture of radical transparency in An Everyone Culture: Becoming a Deliberately Development Organization. As I read about how every meeting at Bridgewater was recorded and available for review by any employee and how employees could rate each other on desirable characteristics such as critical thinking and emotional maturity, I didn’t believe such a culture could be good for people.

I was right.

Copeland describes how Ray Dalio’s theories of society and human development, captured in his “principles” list, governed who was favored at Bridgewater. Dalio’s ultimate vision was to use the data from the peer ratings (“dot voting”) to power an artificial intelligence application that would make human resources decisions on who to hire, promote, and fire. The Fund is a terrifying tale of badly misguided performance management that relied on the opinions of a megalomaniacal founder instead of objective data.

Much of the book describes the multimillion-dollar effort to convert 330+ principles into quantitative measures, which ultimately proved futile. Bridgewater’s efforts reminded me of some HR efforts where I worked with teams to measure abstract concepts like “servant leadership” or “empathy.” This is not to say that intangibles like servant leadership or empathy can’t be used in performance management. I successfully combined servant leadership and situational leadership for effective performance management. The key is to realize the differences between qualitative and quantitative measures.

As state and local governments rush to incorporate AI-powered data analytics into their process, these books provide essential insights into protecting people’s privacy, building safeguards in using data, and ensuring that data-driven decisions are based on good data. AI has much potential for good but beware the dark side.

?