Big Data, or The Uncultural Revolution

In my role as consultant and bottle-washer, I’ve recently been working on two projects which have made me think closely about the world and the way it barrels along, generally dragging us, flailing, in its wake. It’s an unshakeable habit of mine, and I’d apologise for it if I thought that would help.

The first of these projects brought me into contact with a customer who offers a big data analysis service, based on Hadoop. The second is one of our current favourites: a migration from, and subsequent removal of, Windows 2003 Server.

The thinking behind big data processing, as far as I can tell, is that every single transaction, of which there are millions and millions, creates a footprint, some significant data. Whatever the business. I get that. In fact, it’s a concept with which I’m intimately familiar…

Many years ago, I wrote a letter to the wife of an ex-colleague. I had never met her, didn’t know her at all, and I agonised for a long time over whether or not it was appropriate. In the end I sent it, and I’m glad I did. The reason he was an ex-colleague was because he had died, of cancer, at the age of 39, leaving not only his wife but also a toddler daughter. I wanted them to know that, despite being no more than a workmate, there were one or two things (it didn’t matter what they were) that her husband/her father had said or done which remained lodged in my memory. From that point on, I wrote, I imagined that every time I came across the appropriate trigger, I would be reminded of those pearls of wisdom, or those pratfalls, think of him and smile. As it happened, I was right: I still do. What that means is that his life, tragic and too-short though it was, has affected mine, and that of who knows how many others, in a positive way. Something he has done has enriched my existence. At the time, my motivation in pointing that out to his grieving family was purely emotional. That’s invariably the way of it in personal interaction: from a vast sea of experience, individual moments will produce an impact aligned to emotion rather than to reason.

Business is a slightly different matter, although not as different as all that. There are set rules, yes, but there has always been a personal side, too. Making sense of the vast collection of individual transactions requires knowledge but also wisdom, and the hard-nosed confidence to act when the time is right. Instinct plays a big part. You might class it as abstract ‘judgement’. So it lives at least partly in the emotional intelligence space, too. But that judgement is generally applied at another level. It is merely an enabler on the way to an outcome. It must be applied before it has validity.

What am I talking about? Take Spring. The season. What do we know about it? An experienced executive might apply his mixture of rules and judgement: have we had the vernal equinox? are the magnolias in bud? have more than seven of the last 10 days dawned without a frost? have I turned my heating down? does it feel like Spring? In that case, it’s Spring. Let’s get the new season’s lines into the store. Now think about Summer: what colours are going to be big? cotton or linen? where should we advertise? The treadmill never stops.

Robert Browning, a poet, took an alternative approach. Of course he did. His lack of urgent ulterior motive allowed him to stare long and hard at Spring and interpret it through a tableau he was able to paint in his head and through his carefully-chosen words. Spring meant nothing more than what it was, and nothing less than everything it was: chaffinches, whitethroats, pear blossom, buttercups, thrushes delicately balancing, showing off; elms resurrecting themselves. Still a treadmill, but with a vastly different tone.

One isn’t right and one isn’t wrong – I’m not being that floaty and simplistic. Quite the opposite, in fact. I’m saying that the gap between the two is closing. And the reason the gap is closing is that too many opportunities have been missed, and business does not allow that to continue happening. There is competitive advantage to be gained by employing something of the poetic approach.

Let’s face it; fundamentally, we, as human beings, are no different from how we have always been. The Earth still goes around the Sun, we still walk on two legs, we still use complex language forms to communicate; we still get married, ride bikes, throw discus. We still require food and shelter and companionship. By the same token, judgement has never stood entirely alone. It’s very rare that the qualitative insight of even the sagest business guru has been built without the aid of quantitative analysis. Statistics, management information, data mining, data warehousing, business intelligence, big data: call it what you will. The change is in the technology at our disposal and its ability to extract meaning. Now, every tiny transaction can be scrutinised, and not just the transaction itself but its relationship with all the other tiny transactions in orbit around it.

What we are doing, effectively, is living in ever higher resolution. It’s what poets have always done, and the rest of us are joining them. Not because we want to be poets, but because we can. And we must. Those roughly defined zones of knowledge are being zoomed in upon and replaced by millions of tiny data points.

This is a good thing, isn’t it? It’s good for individuals: for those among us without the years of experience and the battlescars our elders and betters might sport. We’re able to demonstrate and exercise levels of judgement that would previously have been beyond our capabilities. But is this just commoditising wisdom? The equivalent of years of graft and pain and hard-won knowledge is now available at the click of a button and the whirring of a few hundred processing cores. Why should life be that easy? No, it’s fine: bemoaning that is like saying that we should never have invented the printing press, or even language itself.

And it’s good for business, too. The Spring line might be doing just fine, but are we making the most of the opportunities of the season? Look closer: look at the whitethroat building his nest. Browning did. Have you noticed: the poor wretch can only carry a single sprig at a time? How difficult would it be to manufacture and market a simple twig consolidation device that could package together three or even four into one journey? What weight of payload can a whitethroat manage over a short distance? Or what about that pear tree, making an unholy mess on the clover? There’s a gap in the market for a blossom hoover. A buttercup scythe? Simple, and life-changing. So it goes on. In other words, we’re all granted the insight once confined to those of a poetic leaning. The geeks have taken over: an uncultural revolution.

There is a cost, too. And that cost is partly played out in the current death throes of Windows Server 2003, the second project I mentioned at the top. Now, by any measure, this is a powerful piece of computer software. Even taking into account what we all know about the onward march of technology, it can perform an impressive array of functions. But that won’t save it: so recently at the forefront of consumer-grade technology, it is now as obsolete as a ZX Spectrum or a John Bull printing set. 32 bits in your processor? A mere 4GB of addressable RAM? Pah! Get lost, shrimp. Even Microsoft have had enough of pandering to their baby and are disinheriting it. That creates work for consultancies like the one I represent, and creates expense for our customers.

None of this is a disaster, needless to say. It’s totally inevitable. We call it progress and we demand it. Is the so-called ‘progress’ simply the digging and filling-in of endless arrays of Adam Smith’s holes, or does it genuinely make the world a better place? Well, probably, the latter. It’s certainly a more pleasant place to live for the majority of us – go back 100 years or so and ask anybody who had polio or TB or had to share an outside toilet with twelve other families, or give birth.

So what’s the danger? I’m no Luddite, but is there a risk that we’re running vast swathes of microchip-driven analysis just to confirm what the wise and/or sensitive amongst us already knew? And, worse, are we removing any vestige of romance? Do I really want an array of worker nodes to tell me what I once went to a poet for? Whoa. I expect I just need to adapt. Poets will also need to do the same, I feel.

I had a dream recently. I’m not sure of its significance in this article, so I’m tacking it onto the end. In this dream I held a piece of paper in my hands. A satisfyingly sturdy piece of paper. Almost card, but definitely paper. Printed upon it were numbers. I knew what they were. They were all the telephone numbers in use in Winchester in 1975. Forty years ago. I lived in Winchester at the time and I was 6 years old. It was a big piece of paper, and it carried thousands of digits, some in groups of four and some in groups of five: we had both formats in use back then. But what I noticed most was the white space between the phone numbers. There wasn’t a whole lot, but space it definitely was. Each telephone number had its own dedicated real estate. I found our number, and those of my friends. I felt peaceful and nostalgic. Then I turned the piece of paper over. On that side were all the numbers in use today. Gone was the white space. Some even overlapped with others, forming an illegible thick blob of black. It was ugly and utterly terrifying, although compelling to the point of paralysis, and after some time I woke up exhausted. Now I’m afraid to even allow the image back into my brain. It’s extremely brave of me to bring it up here.