AI & Node Snarfing

The digital world's era of rapid growth of people and engagement is ending. Now the game is about node snarfing.

Networks = Nodes + Pathways

A network is a system of?

• Nodes = places where something important resides, like a person, business, piece of content, or metadata
• Pathways = connections between nodes

The phrase "All roads lead to Rome" sums up the essence of a network. Across Europe, there are roads (pathways) that lead to towns and cities (nodes), including Rome.?

It could just as well have been "All roads lead to Paris" or "All roads lead to all towns and all cities on the Eurasian megacontinent." But when the phrase was coined, Rome was positioning itself as the global center of power and politics, kind of how Las Vegas tried to position itself as the global center of licentiousness with "What happens in Vegas stays in Vegas."

As illustrated in the image above, my Instagram profile is a node on the network of the Internet. A photo I post on Instagram is a node connected to my profile. That photo post has a bunch of additional nodes associated with it — metadata with details about the post.

The Era of Rapid Growth is Ending

Two-thirds (67%) of the 8 billion people on Earth are on the Internet, according to the International Telecommunications Union (ITU). That leaves 2.7 billion people who have yet to come online.

This very day in November 2023, about one million people around the world, in countries like Indonesia, Pakistan, Cambodia, and the Democratic Republic of the Congo, will decide they have enough disposable income to buy a phone and pay for a data plan. For many of them, it will be the biggest investment they've ever made, but they believe it will be worth it because they've seen people around them buy phones and use them to communicate, learn skills, and be entertained. Today they're becoming nodes in the digital world.

That's a lot of new daily Internet users! However, the global growth rate of the digital population is about half what it was fifteen years ago.

In the countries of Europe, the Commonwealth of Independent States (CIS) and the Americas, between 80 and 90 per cent of the population uses the Internet, approaching universal use (defined for practical purposes as an Internet penetration rate of at least 95 per cent). Approximately two-thirds of the population in the Arab States and Asia-Pacific countries (70 and 64 percent respectively) use the Internet, in line with the global average, while the average for Africa is just 40 percent of the population. (ITU)

For the richest parts of the world, and the richest parts of the poorest parts of the world, pretty much everyone who's coming online is already online.

The era of rapid growth of the digital world is ending. We can all feel it in how our relationships to digital products have been changing. In Stratechery, Ben Thompson has been chronicling this in what he calls The End of the Beginning of the tech industry.

While the digital world was rapidly expanding, companies like Google, Amazon, and Apple and platforms like YouTube, Facebook, Instagram, and TikTok, focused on growth — of users and their engagement.?

It didn't matter greatly to Facebook that time spent on YouTube was growing rapidly, because time spent on Facebook was also growing rapidly. In both cases, the rapid growth was difficult enough for each platform to keep up with — neither company was focused much on trying to win time from the other — they were more worried about the capacity of the Internet growing fast enough to accommodate their own growth.

As the growth period ends, it's becoming more of a zero-sum game for platforms. Every additional minute a person spends watching TikToks is directly at the expense of YouTube or Instagram. This is why YouTube Shorts and Instagram Reels were both rapidly developed and launched — both platforms were losing time to TikTok.

It's common for companies to talk about their market share as "a slice of the pie."?

When the pie that is time spent online was rapidly growing, it didn't matter much if the size of your slice relative to competitors was getting wider or narrower because the whole pie was growing so rapidly that the total volume of your slice kept growing.

Now that the game is more zero-sum, if someone's slice gets wider, it means someone else's slice, maybe yours, gets narrower.?

A big portion of the latest wave of massive tech layoffs is coming from this, as companies shift from focusing on growth — attracting and retaining more people and more of their time — to focusing on optimization — generating the most value from the time spent by the people they already have.?

(Those tech industry layoffs included me: My growth-oriented role at Meta was eliminated in July, and the people from my team who remained were shifted to optimization-focused work.)

The only way to grow the size of your slice is to win some slice from competitors, right? Not necessarily.

"Make the pie higher!"

George W. Bush was known for his malapropisms like, "They misunderestimated me," "I think we agree, the past is over," and "Rarely is the question asked: Is our children learning?" A word was coined to describe them: Bushisms.

One of my favorite Bushisms occurred during a Republican Presidential Debate in 2000, when President Bush was seeking reelection. He said, "What we Republicans should stand for is growth in the economy. We ought to make the pie higher."

"Make the pie higher" immediately went into the lexicon of Bushisms. And I laughed along with the other critics. He clearly mangled "make the pie bigger." But today, I think it's sort of a brilliant description of the current state of the digital world.

There are two ways for a digital platform to gain value from people's time: make its slice of the pie wider by winning more time from people (which is becoming a zero-sum game), and make its slice higher to generate more value from the time people spend on the platform.

That's what the current AI hubbub is about: making the pie higher overall and making each competitor's slice higher even if it's getting narrower. AI helps companies be more efficient (the same or fewer employees can do work previously done by more people), increase engagement (more accurate search results or ad targeting drive more frequent consumer usage and purchasing), and develop new products that weren't possible before (like content made from generative AI models).?

How do you make your slice higher?

Node Snarfing

If you haven't been working in the tech industry or haven't been obsessively learning about artificial intelligence (AI), it largely boils down to this: Since the invention of the transistor, the cost of computing has steadily gone down over the years. In the past few years, it's gotten cheap enough to throw a lot of math at a lot of problems that previously were too expensive to solve.

Imagine a haystack with a needle hidden somewhere within it. You could sort through the hay by hand to find the needle; it probably would take a long time. If a friend helped you, you'd find it in perhaps half the time. AI is like having a million friends help you find that needle, instantly.

AI doesn't just find the needle. It catalogs where all the straws of hay were in relation to the needle in order to make finding the next needle in the next haystack easier and faster.

For an AI, each of those straws of hay are nodes, and the positions of those straws are metadata.

We generally use the Information We Collect…To train and improve our technology, such as our machine learning models and algorithms. – TikTok Privacy Policy (Last updated: May 22, 2023)

The more nodes you have to analyze, the more you can do with it. As the growth era ends, the primary game for digital companies is shifting to node snarfing, collecting or generating as many nodes as possible in order to use them for machine learning and AI products, for improving internal efficiency and for improving consumer engagement and value.

If I make a video on TikTok and then repost it to Instagram, IG collects some nodes from the video. But if I make the video natively on IG, they get even more nodes they don't get from the TikTok repost, such as: camera effects, music used, my location when posting it, etc. They can use those nodes to better personalize content and ads, generating more value per user.

If you purchase stuff from Amazon, and if you have a Gmail account, you've probably experienced this:

AMAZON EMAIL:? Hi Brian. Your package has been delivered.

ME:? Which package, Amazon? I'm expecting two packages. One is a book I've been meaning to read that I finally got around to purchasing. The other is a spare part for my broken kitchen faucet that I desperately need so that I can fix it and turn my water back on. I've been living for two days without water! I'm away from the house right now, but I'll rush back home if it's the spare part. Which one is it???

AMAZON EMAIL:? Your package has been delivered.

ME:? Aaargh! (rushing home to find…it's the book)

Amazon doesn't tell you which package it delivered because it doesn't want Google snarfing their nodes through your Gmail messages.

TikTok's big era of growth in the US was during the pandemic, as buzz about the app spread to people spending most of their time at home. But it had nearly as big a growth era a couple of years earlier, in 2018.?

Throughout that year, ByteDance (TikTok's parent company) spent heavily on app install ads on Instagram, Snap, and Facebook. These ads were cheap and well-targeted. And as a result, TikTok snarfed up users from those other platforms.?

I'll talk more about AI and node snarfing in upcoming articles. While I pay close attention to how my data gets used by digital companies, I'm pretty excited about how node snarfing is resulting in some exciting new products and capabilities, and generally I think the latest worries about AI are overblown. (When I stop getting ads for products I've already purchased and likely won't purchase again for a while, then maybe I'll start worrying about our robot overlords.)

How do you feel about your nodes getting snarfed?

You can buy the t-shirt here . (I don't make any money from it.)

[Disclosure: I'm long META. I still own stock that I received as part of my compensation while I worked there from 2014-2023.]