Algorithms for Confirmation Bias

This is probably more of a rant than anything useful, but I've been meaning to put my thoughts down on paper regarding this topic for awhile.

It started with shopping

I'm sure we’ve all had this, you’re looking for something very particular, lets say I’m looking for a new TV on Amazon, I Google some reviews, I do my research, then I buy my TV, my buying experience is complete. Next day however, my Amazon front-page is full of TV’s, thanks, that might have been useful when I was shopping for one, but I’ve bought it now. I’m browsing the internet and Google Adwords is feeding me TV deals. Maybe I kick myself because I see the same model cheaper, but regardless I made my purchase and I no longer have a need for a TV.

Advertising is becoming more intelligent. I saw this story shared from a friend: they’re member of a pet store points program, and after they bought some puppy food they got very targeted advertising about vaccinations, puppy toys and so on. This is smarter as its following the buying cycle. If I buy a TV maybe I want bluray player, or a sound system, clearly I don’t want another TV. What is happening is building up a persona, a profile for you and targeting you with things you should like based on buying cycles of people with a similar buying profile to you.

This is probably obvious, but it’s important to cover the background in order to introduce the problem.

Article Algorithms

I think every service I use, Facebook, LinkedIn, Twitter, Google, Amazon, all have algorithms that prioritise the feed according to the profile they’ve built up against me. Sometimes this is harmless, I go to LinkedIn and I rarely see Oleg after reporting most of their posts, instead I see stuff about Cloud, Security and so on, stuff I’m (mostly) genuinely interested in. I get the same on Twitter, I follow over 3000 people and I always think it’s amazing that when I look at my Twitter feed there’s the same couple of people seemingly always posting, even though they don’t post that often, but I like a lot of what they post and so it’s nice to see them all the time.

What’s happening is I’m being fed an information bias that reinforces my views. Clearly the whole world is talking about the issues of cybersecurity in the cloud because my LinkedIn feed is full of it. No, I just keep reading and liking articles that discuss this. Clearly Dr Katie Chong is a Twitter rockstar and everyone gets her at the top of their feed, (well of course she's a rockstar, bad example), but I read and like most of her tweets as she’s one of my best buddies, does amazing stuff, and Twitter watches this and tweaks my timeline so I’m guaranteed to always see one of her tweets. So this is fine, so far this has been a benefit, I’ve got some noise reduction and focus on what’s important to me.

Except it isn’t really, it’s what Twitter, LinkedIn, etc. thinks is important to me, against the algorithms that they refuse to (or can’t because they’re so complex and/or auto-learning) explain, so I’m getting a bias based on ‘likeable’ or ‘viewable’ content, which isn’t the same (AT ALL) as content that is important to me. One of the reasons I hate Facebook, my friends that aren't Facebook friendly (i.e. don't produce 'libeable' content), disappeared from my news feed, even though their content was incredibly important to me.

Confirmation Bias

This quickly becomes dangerous, firstly from a business point of view. A fictional example, not a commentary on the technology on either of these scenarios, just useful for illustration:

• I’m a VMware vExpert, I love everything that VMware does and like every article on VMware. I keep getting news about how the whole industry is continuing to grow VMware, public cloud adoption is slowing down, here’s another story about someone moving their workloads out of the cloud because it was 10x more expensive, and here’s a great article on how 75% of workloads last year were migrated out of the cloud. Look here’s an analyst report that suggests the same. I’ve reinforced my views that VMware is the right choice and I’m doing my company the best service by sticking with our current model. Maybe this is true, remember no comment mean on the technology, just an illustration.
• I’m an AWS aficionado, I’ve been playing with it for years, our company runs a significant workload in AWS, has done for years. I see lots of other people doing the same, I see articles about people moving away from traditional, legacy on-premises virtualisation solutions and adopting cloud in a big way. I see articles on the cost savings had from cloud migrations, and I see how AWS has grown year-or-year continually and is now a behemoth of a business. Look, there’s an analyst report that suggests the same. I’ve reinforced my views that AWS is the right choice and I’m doing my company the best service by sticking with our current model. Maybe this is true, remember no comment mean on the technology, just an illustration.

The main problem is it’s becoming increasingly difficult to get an opposing opinion, almost every site I know employs algorithms to tailor news to be ‘interesting’ and ‘relevant’ to you, but none of them have a way of challenging your view. Even at conferences and meetups, they’re tailored to a single vendor or view of the industry, so getting an alternative views is really hard.

I was introduced to confirmation bias several years ago when first getting into digital transformation projects. You need to challenge the business and individual teams with what they fundamentally know is right, their way, with what you believe is right, your way. And this is the source of many problems and issues with digital transformation or modernisation projects:

The Most Dangerous Phrase In Business: We've Always Done It This Way.

That's not an excuse for change for the sake of change either, that's the other side of the digital transformation coin. Maybe it doesn't need to change, maybe everything is perfectly fine the way we've always done it. What you simply need to do is challenge this to make sure it is truly the right choice. This has basically been what my job is for the past 5-6 years now, asking people to question what they do today to see if there is an alternative, potentially better way to improve something, somewhere, somehow.

Fake News!

I hate that phrase, but it’s come to sum up an issue that these same algorithms are creating in the world at large. Let's try take 2 examples without being too specific and trying to make this less regional (some of my references are very UK influenced, but hopefully this translates):

I read a popular newspaper that sells for 20/30p and it gives me all the news in simple to read format without using big words or trying to be clever. It tells me that immigration is a major problem, our government are screwing things up, and the bankers are ripping me off while earning millions from their Surrey mansions. The world is at crisis, but our country are the good guys and that particular country over their is trying to crush our strong industry.

I read a big newspaper that I have to layout on the kitchen table or I can read in the park and look like a spy, it has big words that challenge my intellect and I love the cryptic crossword (although I can never complete them). Recycling is my problem and I’m destroying the world every time I use one-time plastic cups or straws. The government are doing a great job, but are blocked from doing what’s right by the opposition and ‘fake news’ outlets spreading fear. Our countries industry is booming and we need to work with that particular country over there in order to be a success.

In both scenarios:  I read and like things on Facebook and Twitter that reinforces what I’ve just read, and I follow people that write this great content. Honestly I see a lot of great content, so I know the country agrees with these views. I was talking to Chris down the pub the other day and I don’t understand where they get their views from, it’s simply not true and there’s nothing to back it up, I probably will avoid Chris in the future because they’re wrong and it was hard work talking to them.

This is a potentially dangerous confirmation bias, this isn’t just businesses making ill informed decisions, this is voting citizens having a narrow view reinforced.

Now comes the really dangerous part, sponsored targeted content that isn’t clearly shown as such. Not only do I have a narrow view of the world, but now I am being influenced in my views without realising it. I’ll read sponsored content, maybe this further reinforces my view, maybe this shifts my view subtly further to the left or right, and because I spent time reading the article, maybe I liked it, the algorithm persona targeting me has just subtly shifted and I’m now fed more similar views. Gradually I’m having my world views and opinions shifted and influenced, in a dangerous biased way that has been influenced by someone paying to do so.

Unsolvable Problem?

I like writing about stuff that I understand, or that I know a solution to. Today I’m writing about something I don’t completely understand (the algorithms involved are complex and the business models are compelling to continue what is happening today), scares the ghoulies out of me, and I have no idea how to solve. You can work hard to challenge yourself, but that only works when you are conscious of a bias and it's hard work to do. I try really hard to challenge my own confirmation bias, but it’s increasingly difficult to do with algorithms across almost every source of news / information. My children are growing up in a world where opinions are fed to them and original thought is really hard. Gender bias is thrown at them constantly from the point of birth (as hard as my wife and I work to raise gender equality boys, it’s impossibly difficult), even political bias is encroaching on kids as they are encouraged more to use the internet to do research for school projects, which suffers from all the algorithm bias discussed above.

Suggestions welcome: my opinion on possible fixes is that they will only work if they are easy to use. Doing lots of research to get a variety of views is what we have to do today, but it's hard work and is not a reasonable fix when the vast majority of people have 1 or 2 outlets for their entire world view.