Financial Journalism in the Age of Elon Musk’s Dystopia (Part 1 of 3)

On Aug. 1, sports apparel and footwear company Under Armour reported that it would be undergoing an operational restructuring that would include layoffs and cost from $110 million to $130 million for the fiscal year. The stock dropped 10% by the end of the trading day following the announcement.

If you relied on the Associated Press’ coverage, you wouldn’t have known any of that. They covered it - but the story was written by a robot, as are thousands of articles on corporate earnings the AP is presently publishing with almost no human oversight.

AP’s three-year-old robot reporting program is powered by artificial intelligence technology provider Automated Insights, which has a natural language tool that produces readable stories from earnings announcements. The headline for Under Armour's second quarter is “Under Armour reports 2Q loss” and the story leads with this: “Under Armour Inc. (UAA) on Tuesday reported a loss of $12.3 million in its second quarter. On a per-share basis, the Baltimore-based company said it had a loss of 3 cents. The results exceeded Wall Street expectations.”

Nowhere in the rest of the story does it mention the restructuring. Here’s the rest of the AP article in its entirety:

“The results exceeded Wall Street expectations. The average estimate of 21 analysts surveyed by Zacks Investment Research was for a loss of 6 cents per share.

The sports apparel company posted revenue of $1.09 billion in the period, also surpassing Street forecasts. Fifteen analysts surveyed by Zacks expected $1.08 billion.

Under Armour expects full-year earnings in the range of 37 cents to 40 cents per share.

Under Armour shares have fallen 31 percent since the beginning of the year. The stock has fallen 49 percent in the last 12 months.”

So this seems incomplete, but is it worse than human reporters who've up until now been forced to write one too many mind-numbing earnings stories? Not entirely.

A news search on Aug. 1 shows several other articles written by humans, most that do a better job capturing the main news. Yet there's still a mix of quality, most of them were slower, and one is a dreadful effort by what appears to be a now-defunct hyper local San Francisco blog, Rincon Hill News.

So is the AP story better than nothing? That's worth looking at, and in fact there's increasing data and research that is starting to give us a better answer.

What we can say is that both the robot stories and the human business coverage leave much to be desired, especially in an era of increasing complexity and information overflow. To say the best result is always going to be the combination of the two seems obvious - but it's so rarely done right. It takes the most technically skilled reporters using the most cutting edge technology to do that, and that's - fortunately and unfortunately - almost all going to be behind a paywall first (but not forever), which I'll get into later. In this three part series, I will explore first the full automation of things in business reporting and the benefits and limitations therein, and then the examples of business journalism that are more human led, and then the best way to combine human skills and technology in this area.

What Can We Fully Automate

First to the impact of full on automation - which the AP has provided us with a unique opportunity to observe.

Automated Insights describes the relationship with AP as, “The Associated Press uses Wordsmith to transform raw earnings data into thousands of publishable stories, covering hundreds more quarterly earnings stories than previous manual efforts.”

A study from November of last year done specifically on the Automated Insights earnings stories by researchers at Stanford University and the University of Washington appears to provide evidence of an increase in trading volume and liquidity after the automated articles come out, but no real impact on price discovery.

So based on this study, people are trading on these stories and likely nothing else!

To me, that seems alarming, but the study found it positive. One of the conclusions was that "because basic synthesis and dissemination of public information via automated articles appear to have capital market benefits, future technological advances allowing for automated interpretation and investigative content have the potential for even greater market impacts."

But what the study does not explain is what people are taking into account when they are trading on just these articles.

The AP boasts a membership of about 1,400 daily U.S. newspapers and thousands of television and radio broadcasters. Its website says more than half the world's population sees AP content every day via 15,000 news outlets, and it operates out of 263 locations in more than 100 countries producing 2,000 stories a day, 50,000 videos a year and 1 million photos a year.

Its Automated Insights site says before its business journalists could only produce 300 earnings stories per quarter. Now, with robots, AP “produces 3,700 quarterly earnings stories – a 12-fold increase over its manual efforts." So the AI juices those numbers by 13,600 stories a year - less than 2% of AP's story output. I can see how business reporting isn't their forte, but it's not as if they were dying for content.

And here's where the AP gets a little carried away, claiming, "The stories retain the same quality and accuracy that readers expect from any of AP’s human-written articles. Aside from an explanatory note at the bottom of the story, there is no evidence they were written by an algorithm” (emphasis mine).

Here’s my own very unscientific breakdown of the 15 recent Automated Insights headlines from earnings between Oct. 10 and Oct. 13 - let me caveat that earnings stories aren't supposed to predict stock movements, but since the Stanford/UW study suggests that people are trading on just the AP stories themselves - which have basic positive or negative indicators - and no other information, I wanted to see how helpful that might be:

First, look at those headlines. I'd have to disagree that there's “No evidence they were written by an algorithm.” But that doesn’t really matter; business reporting can be dry and formulaic, so I’m still on board. But the rate at which the thrust of the article predicates the direction of the stock isn't much better than a coin-flip.

Automated Insights spokeswoman Laura Pressman provided a statement standing by how their work has reduced errors and increased productivity: "We produce earnings announcements for the Associated Press based upon data that is included as part of a company's quarterly earnings filings. This work has helped to reduce errors in both grammar and data content as compared to the hand-written reports that they produced previously. These announcements are intended to provide a factual recap of a company's earnings data and contain no forward-looking analysis or predictions as to the future value of a company's stock," Pressman said in an email. Alos, an Automated Insights engineer - who is a Goldman Sachs alum - helped me with some of the reasoning behind the program.

Yet if there’s any significant price move, the stories in this sample have missed the reason. Blackhawk, for one, dropped 20% not on posting a “3Q loss” but based on disappointing growth expectations in the future - which the rest of AP’s automated article does not report. 

These earnings stories can often be the only public articles reporting on some companies’ earnings; the Stanford/UW study found the increase in "abnormal trading volume and depth" in a sample of 2,249 companies "with little previous media coverage" (note: the sample was early in AP’s roll-out, I was to find out, ending in late 2015).

One example of a company with little previous media coverage is Oil-Dri, where the AP's article was also missing some pertinent information.

The AP’s headline, “Oil-Dri posts Q4 profit” ignores that the profit in the fourth quarter is actually down 75% year-over-year, among other things. The stock plunged 17% the first day of trading after the announcement.

And the AP stories don't include other major factors that give an indication of post-earnings performance. These are factors that could be automatically included, such as the stock performance the two weeks prior: Goldman Sachs' internal research from early 2016 suggested that stocks that underperformed in the two weeks ahead of the event tended to have stronger positive reactions on earnings day.

So it could seem people trading on just the AP's stories are going to be at a disadvantage - but that's not necessarily true either, according to the Stanford/UW study's authors.

I spoke with one of the authors, Ed deHaan, assistant professor of UW's Foster School of Business, who collaborated with Stanford Associated Professor Elizabeth Blankespoor and Christina Zhu, a Stanford PhD student.

deHaan says the increased trading presumably comes from people who are seeing the automated stories and then somehow deciding whether or not to buy or sell. However, the study does not look at what exactly from the stories these people are trading on. That is the focus of future studies.

deHaan notes one indication that the people who are trading on the stories are not making worse bets than the market in general - because there was no evidence of increased short-term volatility. Trading was later absorbed by the market makers and brokers without impacting the price path, he says.

"We didn’t find any evidence that people are buying at the wrong prices," deHaan says. "It seems like the prices they’re buying or selling is roughly where it would have been even in absence of these stories."

However, the study isn't claiming that volume is good by itself. Elsewhere there is a lot of evidence that people trade too much, according to deHaan. And even the utility of extra liquidity is not made explicit, and might not outweigh additional trading costs.

"There is a risk - a newspaper article comes out - a retail investor sees this and decides to make a trade they might not have otherwise made - they could be incurring trading costs unnecessarily," deHaan said.

Ultimately, all of the people involved with AP and Automated Insights and the study did not provide evidence to counter one potentially negative conclusion: Since individual investors almost never beat the market, the added costs of more trades could wind up having a negative financial impact for the already disadvantaged retail investor.

But I mean honestly, if individual investors don't know they're not going to win by now, it doesn't seem like the robo stories are doing much more harm. And one non-financial positive about the stories deHaan posited is that some people may just like to read stories about local companies: there's "infotainment value."

When reached for comment, AP spokeswoman Lauren Easton provided an emailed statement: "AP’s automated corporate earnings stories present factual financial information about approximately 4,000 U.S. companies each quarter."

In the face of the development of fully automating earnings stories, many journalists welcome it as a development that reduces the grunt work. It means more time working on breaking news, investigative news, and more nuanced detailed stories. I'm among those. My main concern here is not that about generating content like this automatically - it’s to publish it before there’s any utility. AP and Automated Insights are making the case that the increased volume and liquidity demonstrate that utility - but I still hesitate to promote publishing stories that highlight the wrong things or ignore major announcements.

The AP isn't the only company fully automating earnings reports by any means. If you have any brokerage account, you can get a ton of automated reports of questionable utility. But AP has provided us with the most data to see what impact these reports are having - and so far it's subtle but tangible.

Niche Auto-Reporting

Where there's not as much clear data is using AI efforts to fully automate coverage niche markets. One of the first places I heard about people who were actually peddling AI in niche business news - and making money off of it - was at a conference for the Specialized Information Publishers Industry. NewsRx seems to have been around the longest, founded in 1984 by C.W. Henderson who started a newsletter on AIDs research, and later in the 90s started to try automating the process.

But at the conference I attended in 2009, the executives presenting were fairly grandiose in their claims, essentially predicting the end of the need for humans to do any reporting (Elon Musk would have loved it!). And that’s consistent with how the company presents itself on its website, which says Henderson developed “technology that would revolutionize the way the world thought of research journalism” in the form of “artificial intelligence writing technology that automates every aspect of news creation, from research, writing, and editing to design, layout, distribution, and analytics.” Like the AP, the numbers are impressive. According to the company’s SIPA profile, NewsRX is “the world's largest producer of professional news and write (sic) more than 10,000 original articles each day.”

Even in a recent press release, management has been making big claims about the progress of AI in journalism. Henderson states in the release that, "Three research studies from 2014-2016 confirm that computer-written news tends to be rated higher than human-written news in terms of credibility. Readers find texts generated by machines to be more trustworthy and accurate, especially when they do not know that algorithms were used to do the writing."

The company has not responded to my requests to specify which studies those were (The Stanford/UW study does not get into credibility).

NewsRx claims to produce 195 weekly newsletters on everything from health, pharma and life science to business, tech, energy, law, and finance. The list of clients includes media and research organizations like Thomson Reuters, Dow Jones, ProQuest, Westlaw, and Cengage. But I’ve searched for any big scoop they’ve gotten or even oblique references that they claim on their website in major media outlets - and even the quotes on its website that the New York Times commented “The World’s Largest” and the WSJ provided the equally vague comment “Extensive”, do not appear to be commenting on NewsRX itself (I searched and am a subscriber to both).

I have not yet heard back from the company with requests for links to these specific articles.

I don’t doubt there are some references to one of their nearly 200 newsletters, or one of their staff did something important that was reported on. Either way, it doesn’t seem that they can claim many impactful pieces of reporting over their 33 years of business.

Still, grandiose claims and self-promotion for these "revolutionary" products are pretty standard for startups (The show "Silicon Valley" nails it in this parody of a startup conference). The key here is, neither in broader B2C business news nor in niche B2B business news has full automation really produced what I would call good journalism - though in both cases there's clearly something appealing and it would appear money-generating in doing them.

Speed Up, Not Replace

So AI doesn't produce Pulitzer material quite yet. The sweet spot is much more subtle and was identified well this July in the Harvard Business Review by Walter Frick.

The article explains how auto-summarization techniques usually fit into one of two categories: extractive or abstractive - and then gives an example of each trying to summarize a 5,000-word article compared with a human. The extractive method was done by Fast Forward Labs, a research firm that trained a neural network to score sentences according to the likelihood that they would be included in a summary. And the abstractive was by Alexander Rush, an engineering professor at Harvard who trained a computer to write three-sentence summaries of CNN articles.

They're worth reading for yourself, but the robot articles in my view give the reader a skewed idea of the original article, getting some main points but missing the mark in general - it didn’t seem like either method produced a thoughtful summary of a complex feature.

But Frick makes the point that right now the question is not whether computers can replace humans, but rather “...whether AI-written first drafts of summaries might speed up our process. And here the answer is almost certainly yes.”

That’s the right way to look at it at this point.

There are right ways and wrong ways to combine technology and journalism. In part two of this series, I will get into how to get that balance right for business journalism - and why the places best primed to take advantage of all the advances are not the major existing business news outlets - both for cultural and structural reasons. I will go into how technologies without content to show their utility are worse off than quality content without technology, as evidenced by new news organizations like Axios and The Information, which have shown success largely without a Saas component by embracing pure journalism.

We’re learning - not just as consumers, but also those of us who work in media organizations - the best business reporting you have to pay for. And my experience with niche reporting has been that small, focused teams will beat the behemoth media organizations nine times out of ten despite the disparity in resources. The increasing complexity and specialization in business, combined with the democratization of the most sophisticated technologies, has provided an opportunity for nimble teams to compete with major news outlets who don’t act with enough cohesion or dedication to a particular space. More often, some of the best future business journalists are going to start their professions with some other technical expertise and come to journalism later - and have it not be that drastic a pay cut. The technology in turn will cater to these elite teams - as a starting point, something that will arm them best to dig up stuff before anyone else, but that will never replace them.  

Elon Musk keeps on telling us artificial intelligence is the biggest existential threat to humans. His latest warning at a meeting of American governors in July was particularly dire: “...until people see like robots going down the street killing people, they don’t know how to react...”

“Robots will be able to do everything better than us, I mean all of us,” he predicts.

I'll get into how those of us in the financial news business don't need to worry about that any time soon in part 2.

(Next: Is There Anything Unmeasurable in Good Business Reporting?: Part 2 of 3)