Twitter Makes Life Hard for Marketing, Communication and Research

RIP Twitter API

Elon Musk is always good for a surprise. This time, it primarily concerns market and social research as well as communication controlling – and related software solutions such as media monitoring or engagement tools.

As of today, the Twitter API (Application Programming Interface) is no longer freely accessible -

and thus automatic access to the content published on Twitter and the reactions to it is severely restricted.

Twitter has been the most open network with almost entirely public content and a free, high-performance interface (API). However, despite Elon Musk's official commitment to maximum transparency and free flow of information, Twitter announced on February 2 that it would close the programming interface just one week later. In typical Musk fashion, it is still not clear what will happen next, even on the day of the closure. In the future, access to an automatic evaluation of Twitter content and activities will only be possible for a fee - and possibly only to publish content there, but not to evaluate contributions published on Twitter. But models and prices do not yet exist.

This drastically affects tools popular in marketing and social research, but also social listening applications used in marketing and communications departments.

Twitter follows Meta and LinkedIn on to way to a closed shop

Unfortunately, Twitter's change in direction is not alone in the social media landscape. Facebook, for example, has already increasingly restricted its API since 2018. Without a login, fewer and fewer posts and pages that are actually published as "public" can be accessed. This was initially justified with GDPR, but this explanation is not plausible because the same data is still sold by Meta to advertisers.

LinkedIn (Microsoft) has severely restricted the analyzability and export of data and content from the outset and makes it very difficult for third parties - even from research - to legally access the information published in the network. This is true even where it has been set as public by users. However, the same data is sold at a high price for sales and recruiting (Sales Navigator and LinkedIn Recruiting, respectively).

Privacy and data protection are not the primary concerns of any of the three networks.

Rather, these social media platforms have an interest in slowing down the free flow of information to keep content and data exclusively in their own cosmos and thus monetize the content generated by their users.

"Information wants to be free" – and not monetized

The Internet began its triumphant march with the battle cry "information wants to be free" – and that battle cry resonated for a long time. However, the trend is going in a different direction: every U.S. social media platform is closing its ecosystems and trying to make a cross-platform evaluation that enables comparability and competition more difficult while maximizing its own monetization of user-generated content. "Information wants to be exclusive" seems to be the new motto.

But you don't have to put up with that if you're technically resourceful:

Content that is provided publicly by its creators may in principle also be perceived publicly - whether by humans or bots.

So, with intelligent web crawlers and web scrapers, content on any platform can be evaluated in principle, just as it can be done manually. However, most large monitoring providers are not (yet) able to do this because they heavily rely on interfaces (API) and shy away from the technical effort and the high data volumes of web scraping. At Hase & Igel, on the other hand, we have been relying on our own, independent collection of public content and profiles within the legal boundaries of GDPR for some time now. Our AI-powered applications thus continue to guarantee a good evaluation of content and activities, including full-text search and context classification. These include our Big Data analytics tools Neutrum AI Targeter and Neutrum Topic Navigator, among others.

Grey zone or regulation - who owns published data?

With the analysis of publicly available information, however, a legal grey zone opens up.

While scraping public content is legal, one enters the danger zone when a site or platform stores in its code that it does not want to be read by machines.

The same applies to corresponding statements in the terms and conditions. At the same time, however, the question arises as to whether a platform may decide this at all if the creator of the content in question – the user – explicitly states that the published content should be seen by all people. A specific opt-in or opt-out for content creators as to whether their content may also be crawled (whether by Google or by Hase & Igel) is not yet offered by any platform.

This question is highly relevant, as social media data has become indispensable in marketing, communication, market and social research. From the stock market to forecasts of birth rates, from reputation management to addiction prevention, from the fight against fake news to the tracking of pandemics – and of course also for the control of one's own communication – researchers, companies and institutions use social media as an indispensable pulse gauge.

Conclusion: "Wild West" mentality must come to an end

Freedom of information and data flow versus the exclusive enrichment of Big Tech to the disadvantage of the data community – this conflict will continue to keep us busy, first and foremost the legislators and the courts that should act urgently to create clarity for all. So far, EU initiatives have focused exclusively on the protection of personal data and the copyrights of professional journalists and artists. Both are undoubtedly important concerns, but freedom of public information, fairness in monetization, and fair competition on the (social) web must finally receive equal attention!