China’s Ticking Time-Bomb.

It should now be clear to even the casual observer that China has been spying on us for years and stealing reams of intellectual property from our Universities, Government Agencies, Research Labs and start-up companies.

In addition to its blatant Internet hacks and the thousands of student-agents planted within our educational system, the Chinese have managed to participate in venture capital investing through their network of VC firms fully established on Sand Hill Road in Menlo Park, CA, where they routinely take a position sizeable enough to clear the SEC threshold on IP sharing so that its limited partners (LP = Investors in the Venture Fund) will have complete access to all of the startup’s intellectual property.

With the recent COVID-19 pandemic, China has found itself cutoff from much of the Western world’s trace channels and a sudden and shrinking demand for goods manufactured in China has pushed the Chinese economy into a deepening recession.

To the hundreds of millions of Chinese living in the People’s Republic, a shrinking economy is both surprising and frightening. On top of the cessation in demand, comes increased sanctions and tariffs from the U.S. and some western allies. Most residents have not experienced a reversal of fortune like the one playing out now on the mainland in their lifetimes. Frightened citizens lead to an unstable one-party socialist republic which leads to revolution and citizen activism.

As a clear message to Hong Kong, and perhaps an even clearer message to its citizens on the mainland, the People's Republic authorized the Standing Committee of the National People’s Congress – China’s rubber-stamp parliament – to draft a national security law for Hong Kong, bypassing Hong Kong’s legislature and directly violating the agreement signed between China and the UK when Hong Kong was handed over in 1997.

Two Countries, One System.

The effect is to eliminate the rule of law and civil liberties currently enjoyed in the city, and ultimately, kill the “one country, two systems” framework that Beijing has touted to integrate Hong Kong into the mainland and compel Taiwan to move towards unification. The new law would prohibit treason, secession, sedition, subversion and the theft of state secrets. And it would legitimize the presence of China’s state security apparatus in the city.

Many Chinese scholars have suggested that the economy must be much worse than is being reported as such a move by China poses substantial international risk and would only have been undertaken if the growing dissatisfaction by the Chinese citizenry had not reached noticeable levels.

I am no China expert, but my 6 years dealing with Chinese Cybersecurity counterparts in Beijing and Shanghai have taught me that what you see is never what you get.

For Beijing, this move is a public acknowledgement of its inability to resolve the political unrest in Hong Kong without resorting to violence, and that the ongoing protests could ultimately undermine its own national security.

It is also a sign that Beijing has lost patience with the “one country, two systems” approach to slowly incorporating Hong Kong into the fold and providing a road map for Taiwan’s eventual unification with the mainland. As Taiwan has drifted further away from Beijing’s overtures in recent years, the Chinese government has felt less obliged to keep up the false window dressing in Hong Kong and their strategy has shifted from winning hearts and minds to imposing fear – much more comfortable for the current regime.

Beijing is counting on Washington and its allies to come to the realization that hurting Hong Kong would not be in their own economic interests and that act this will give rise to an eventual backing away from their threats to take action. The Chinese act is both a constitutional and diplomatic crisis for Hong Kong, and the West with a very real chance that Beijing may have miscalculated their gamble and the US and its allies will retaliate with economic and other punishments.

Because the Chinese leadership cannot back down and be seen by its governed as giving in to external and particularly U.S. pressures, China is now on a hard collision course with both Hong Kong and the West and suggests that Beijing is now more determined than ever to fight a new cold war with its western adversaries.

One that increasingly looks like it will be fought in Cyber-space.

The Story That Won’t Die.

Lost amid the drama playing out in Hong Kong and the mounting evidence pointing to Chinese research laboratories as the source of the Pandemic, is the continuing saga of the deterministic spying and supply chain corruption campaign the Chinese PRC have been conducting over the past several years.

The story that was well-documented by top-ranked journalists which suddenly disappeared from Bloomberg news once the media mogul’s attention shifted to national politics.

It is instructive to remind ourselves of the facts as they have been reported, as it frames the threat of a serious Cyber-war with China in the proper light.

Three years ago, Amazon began performing due diligence on a startup called Elemental Technologies. The monster of the distribution magic was considering Elemental as an acquisition to boost their streaming video service (now Amazon Prime Video). Elemental made software for compressing large video files and they had several national security contracts that made Amazon comfortable with cybersecurity considerations and the integration with AWS. Amazon has many large contracts with agencies like the CIA.

A third-party contractor discovered some troubling issues on their first diligence pass which prompted Amazon to take a closer look at the hardware servers that Elemental customers installed in their networks. It turns out that these servers were assembled by Super Micro Computer Inc., a San Jose-based company (aka, Supermicro) that is also the world’s biggest independent suppliers of server motherboards.

At the evaluation site, the testers found a tiny microchip, not much bigger than a grain of rice, that wasn’t part of the boards’ original design. Amazon suddenly realized that these servers were probably installed at Federal agencies as well, and after notifying the Federal government, discovered that Elemental’s servers were networked throughout the Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships.

And Elemental was of course just one of hundreds of Supermicro customers.

During the ensuing probe, investigators discovered that the chips were designed to allow attackers to create a stealth doorway into any network connected to the tricked-out servers. Not surprisingly, investigators have traced the motherboards back to factories run by manufacturing subcontractors throughout China.

This attack scenario is far deadlier than the software-based back-doors that we all have grown accustomed to seeing. In fact, a well-done, nation-state-level hardware implant would be like wandering badge-less into the Pentagon unchallenged. Yet here we are with maybe 80% of the world’s servers sporting a motherboard with the additional chip. Completely beneath the radar.

Long Term Access.

There are two ways for spies to alter the guts of computer equipment. One, known as interdiction, involves manipulating devices as they’re in transit from manufacturer to customer. This approach is favored by our own U.S. spy agencies. The second method involves seeding modifications into the core processors at birth.

China makes 80 percent of the world’s mobile phones and 90 percent of its PCs. To pull off a seeding attack would mean that the perpetrators would have to develop a deep understanding of a product’s design, then be able to manipulate components at the factory, and finally be able to ensure that the doctored devices would make it through the global logistics chain to the desired destination. All of which is as difficult as it sounds.

Yet, that is exactly what was

found here.

The chips had been inserted during the manufacturing process, by operatives from a unit of the People’s Liberation Army and their goal was clearly to gain long-term access to high-value corporate secrets and sensitive government networks.

Which they have done.

The chips were designed to be as inconspicuous as possible, appearing more like signal conditioning couplers, another common motherboard component, than microchips, and so they were unlikely to be detectable without specialized equipment. Depending on the board model, the chips varied slightly in size, suggesting that the attackers had supplied different factories with different batches.

Many different factories with many different batches.

The primary role of the chips was to manipulate the core operating instructions that tell the server what to do as data moves across a motherboard, and at a crucial moment, allow it to effectively edit the information queue, inject its own code or alter the order of the instructions the CPU was meant to follow.

The chips were then able to tell the device to communicate with one of several anonymous computers elsewhere on the Internet that were loaded with advanced code; and to prepare the device’s operating system to accept this new code. The illicit chips could do all this because they were connected to the baseboard management controller that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.

This system could then let the attackers alter how the device functioned, line by line, however they wanted, leaving no one the wiser.

As one example, consider the Linux operating system where code resides that validates a user by verifying a typed password against a stored encrypted one. An implanted chip can alter part of that code, so the server won’t check for a password any more, and voila, the secure machine is open to all users, authorized or otherwise. That same chip can also steal encryption keys for secure communications, block security updates that would neutralize the attack, and open new pathways to the Internet.

Apple, AWS, NASA, CIA: No Problems Here.

So far, the investigation had concluded that over 30 companies have been affected, including a major bank, large government contractors, and one of the world’s most valuable companies, Apple. Apple has been one of the top Supermicro customers and had planned to order an additional 30,000 of its servers for their new global network of data centers. But in the summer of 2015, Apple conducted their own investigation and discovered the rogue chips which led to an immediate cancellation of that order and severance of the business relationship.

The extent to which Apple’s existing servers are affected and how those networks have been compromised is unclear. Apple’s official statement held that no networks or servers had been compromised and nothing was at risk.

In 2009, Elemental announced a development partnership with In-Q-Tel Inc., the CIA’s investment arm, along with a deal that enabled Elemental servers to be used in national security missions across the U.S. government. These included the Department of Defense data centers, to process drone and surveillance-camera footage, Navy warships to transmit feeds of airborne missions, and inside government buildings to enable secure videoconferencing.

NASA, both houses of Congress, and the Department of Homeland Security have also been and are still, customers.

Supermicro, founded in 1993 by Charles Liang, a Taiwanese engineer was an early beneficiary of the contract manufacturing outsourcing trend in the Silicon Valley at that time and started his company by providing engineering design services along with access to supply chain factories in Taiwan and China.

A Mega-Monster.

Today, Supermicro sells more server motherboards than anyone else on the planet. It also dominates the $1 billion market for boards used in special-purpose computers, from MRI machines to weapons systems. Its motherboards can be found in made-to-order servers for banks, hedge funds, cloud computing providers, and web-hosting services, throughout the world.

Its core product, the motherboards, are all manufactured by contractors in China. Imagine a sea of millions of ticking time bombs distributed throughout the entire economy, governing agencies and defense systems of the western world waiting at the ready for instructions from their C&C servers in Beijing.  

Hundreds of full-time Taiwanese or Chinese engineers with overseas ties work at Supermicro’s San Jose facility where Mandarin is the chosen spoken language supplemented by the written Hanzi which makes it easier for visiting Chinese partners to gain an understanding of their operations and designs. The Bloomberg investigation was in the process of trying to determine whether active spies were planted inside Supermicro and/or any of its suppliers when it was promptly suspended.

My own sources tell of a locked server closet with a mysterious optical fiber QKD network connection. And no one has access.

The easiest way to imagine Supermicro is to think of them as the Microsoft of the hardware world, with more than $3.2B in revenue, 1,000 customers in 120 countries by 2020, Supermicro potentially offers inroads to a massive collection of sensitive targets. Attacking their motherboards is like attacking the Windows install base.

Horses And Barns.

The reality is in fact that the security of the entire global technology supply chain has been compromised, though consumers and most companies don’t know it yet and those who accept the possibility that the Bloomberg story was based in fact on fact, point to the changes Supermicro has implemented following that publicity.

That Bloomberg story was so credible that is caused Supermicro’s share price to fall 50% and resulted in a de-listing from the NASDAQ.

Jordan Robertson and Michael Riley are two of the most respected journalists in the news business and in spite of being awarded a pwnie for the "Most Over-Hyped Bug" by the security community at the BlackHat USA conference in 2019, Bloomberg not only continues to stand quietly by their story, they have promoted Michael Riley to oversee all of Bloomberg's technology security coverage and have allowed both reporters to continue their research toward discovering the one missing element that prevented this story from surpassing Watergate: evidence.

This lack of evidence has created two camps within the Cybersecurity community. Those who believe the threat is real, and those who believe the official statements by the mega-companies and agencies affected.

Today, Supermicro has transitioned most of its contract manufacturers (CM) to Taiwan and points to this fact as evidence of a de-risking of the Chinese threat. But the same CM management teams that were embedded in the old supply chain remain embedded in the new Taiwanese supply chain, but with new and different Taiwanese employees than their former Chinese counterparts.

Way back in 2014, U.S. intelligence officials had specific and concrete evidence that China’s military was preparing to insert these rogue chips into Supermicro motherboards bound for U.S. companies. But, issuing a broad warning to Supermicro’s customers could have crippled the company, (still a major American hardware maker), and it wasn’t clear from the intelligence exactly whom the operation was targeting or what its ultimate aims were. Plus, without confirmation that anyone had been attacked, the FBI was limited in how it could respond.

So, we did nothing.

In 2018, Supermicro had three primary manufacturers constructing its motherboards, two headquartered in Taiwan, one in Shanghai and subcontract manufacturers scattered throughout China. Four of those subcontractors are now known to have been building out Supermicro motherboards for at least the last two years. Those subs operate the way all subs operate in China.

Plant managers at the subs are approached by people who claimed to represent Supermicro or who held positions connected to the Chinese government. These middlemen would request changes to the motherboards’ original designs, offering bribes where they encountered resistance or suspicion and even threatening factory managers with inspections that would surely shut down their plants. Once these “arrangements” were finalized, they would organize the rest of the supply chain back to Supermicro.

And we know all of this, yet we happily continue to do business with them anyway. Today.

In 2020.

Ignoring China’s Ministry of Foreign Affairs ridiculous statement about China being “a resolute defender of cybersecurity”, we should congratulate instead the current administration’s recent decision to pull all off-shore manufacturing back to the States. China comically wants the world to believe that their 2011 proposed international guarantees on hardware security was somehow meaningful and that their plea to “parties” making “gratuitous accusations and suspicions” must end, so that we can all Kumbaya and “conduct more constructive talk and collaboration together in building a peaceful, safe, open, cooperative and orderly cyberspace.” Right.

A Gordian Knot.

As far as Amazon is concerned, their security team, conducting its own investigation into AWS’s Beijing facilities found altered motherboards throughout those servers, including more sophisticated designs than they’d previously seen. In one case, the rogue chips were thin enough so that they’d been able to be embedded between the layers of fiberglass onto which the other components were attached.

While China has long been known to monitor banks, manufacturers, and ordinary citizens on its own soil, and the main customers of AWS’s China cloud were domestic companies or foreign entities with operations in China, the fact that the country was conducting those operations inside Amazon’s cloud presented a Gordian knot.

The dilemma was that it would be difficult to remove the equipment without alerting interested parties and that, even if they could devise a way, doing so would certainly alert the attackers that the chips had been discovered. So, they instead developed a method of monitoring the chips and in so doing, could not identify any traffic among the entry points and the attackers. Which only told them that the bad guys were saving the chips to mount a delayed attack or that they’d infiltrated other parts of the network before they started monitoring, neither of which was particularly reassuring.

The point is that this is all going on right now while the world ponders what to do.

When two of its major customers, Apple and Amazon bailed out on Supermicro in 2016, its CEO, Liang vaguely blamed it on pricing with the statement, “When customers asked for lower price, our people did not respond quickly enough.” Whether that sort of dissembling raises the question of complicity, I will leave to you, but it seems to me that Supermicro is making it harder to take seriously the company’s case for innocence.

Because concurrent with the malicious chips’ discovery in 2015 and the investigation unfolding around it, Supermicro suspiciously missed two deadlines to file quarterly and annual reports required by regulators and was delisted from the NASDAQ as a result. This seems like extraordinary behavior for a company growing at the breath-taking rate of 100% over the last four years ($871 million in the second quarter of fiscal year 2020).

Xi Jinping Promises to Play Nice.

In 2015, then President Obama and Chinese President Xi Jinping appeared together at the White House for a joint press conference touting a landmark deal on cybersecurity.

China had agreed that it would no longer support the theft by hackers of U.S. intellectual property to benefit Chinese companies.

But anyone who had been paying attention, including senior White House officials knew that China was happy to offer this concession because it had already developed advanced forms of cyber-theft supported by its monopoly of the Asian technology supply chain.

U.S. government officials and U.S. technology executives subsequently met in a Pentagon sponsored event to determine ways in which the U.S. could develop commercial products that could detect hardware implants. Everyone present knew that the hardware on topic was being produced by Supermicro.

Why, in the name of national security did we not simply shut the company down?

I am acutely aware that it is unbelievably difficult to make decisions and execute any sort of rational plan within the labyrinth we call the Federal government. There are hundreds of thousands of people in roles where they have been since the last several administrations, continuing to do the work that they are chartered to do regardless of who is in charge or what their political agenda happens to be. There are tons of moving parts and hundreds of shadow agencies and agreements dependent on any directional movement, one way or the other.

In short, it is not easy to get “easy” things done.

That Dog Don’t Hunt.

Decades ago, we had made the decision to send advanced production work to Southeast Asia and low-cost Chinese manufacturing had come to underpin the business models of many of America’s largest technology companies. Apple, for instance, made many of its most sophisticated electronics domestically up until 1992 when they closed their state-of-the-art plant for motherboard and computer assembly in Fremont, Calif., and sent that work to China.

The assumption that China would surely not do anything like spying in its factories to jeopardize its position as the workshop to the world became the guiding principle for continuing the folly. We seem to be intoxicated with this devil’s bargain as our somatic capitalism continues to opt for the supply chain we desire, even with all of the known global risks versus accepting a lower capacity supply chain with greater security.

Well, COVID-19 just declared that this dog no longer hunts.

The cheap chip ship has finally sailed, though it leaves behind more than 90% of the world’s servers infected with an incurable cyber-disease.

What we do about this next matters more than anything we have ever done in the past.