War?Story: “For?never?was?a?story?of?more?woe, than?of?this?Server?and?its?ILO”

A tale of errors concealing errors, and an Incomplete Network Diagram resulting in Catastrophic Security Failure.

Statements of the Bleedin' Obvious

Several times I have had to write short documents as guidance or policy which amount to no more than Statements of the Bleedin' Obvious. It is always frustrating to have to write one of these, and all the more so for it being necessary.

Once such document described what a supplier (or whoever was responsible) should include on a network diagram in the assurance documentation for “a system”. It was very short; two pages of A4. Then once I'd removed the more colourful language that had flowed so easily as a result of it even being necessary to write such a document, it was just a single side of A4. With margins and white space. Because really, it should not need to be spelled out:

• If any equipment is connected to the same network segment as “the system", then it MUST be included on the diagram.
• If any equipment is separated from “the system” by an assured barrier, then it does not HAVE to be included on the diagram - that is, if some form of security device (such as a firewall) which performs a Security Enforcing Function that has assurance through independent testing of the security device and its configuration, isolates the equipment from “the system”.

I would dearly love to write about the worst ever example of this, but despite the passage of significant time it is still far too sensitive a matter. So instead, a concrete example of how failing to follow such a basic and obvious approach led to a Catastrophic Security Failure - luckily one that was found by a CHECK test rather than anyone of a more black-hatted tendency.

Integrated Lights Out Management

This comes in a variety of flavours and brand names, but they're all fundamentally the same sort of thing: some hardware which gives you some of the same access to a computer that you would have with direct physical access. You can power cycle the device, provide direct keyboard/mouse input, see the video card output - all remotely.

They plug into the network, but of course that would be the management network not the same network segment as the computer itself, of course. That would be Really Dumb?. The ILO provides a path of highly privileged access.

Physical access to a machine give you Special Powers?. It usually lets you defeat much of the security around the system and OS - which is why we keep these machines locked away securely in data centres with big walls around them and guarded by ex Sergeant Major types[0]. The heart of a Data Centre, the Citadel, is usually devoid of people most of the time, so the lights can be switched off to save power. Hence “Lights Out”.

For the avoidance of doubt, hypervisors typically provide this type of access to their virtual machines.

CHECK Testers Are Your Best Friends

When a CHECK team manages to own your system and demonstrate how they could steal all your data, they are saving your from considerable embarrassment and worse[1] - provided, that is, you take the issues raised and fix or otherwise mitigate them[1].

This makes them the best friends you will find. Treasure them, respect them, and listen to them.

In fair Verona[2], where we lay our scene....

What went wrong and why, and what did it all mean?

What Went Wrong?

• The Lights Out (ILO) capability was not included in the network diagrams or even mentioned in the assurance documentation.
• Therefore, the capability was not included in the risk analysis or considered in the risk management.
• The network topology of the ILO capability was not subject to any scrutiny. Difficult to scrutinise that which you do not know about.
• ILO hardware was plugged directly into the internal network. This is a n00b-level error.
• The ILO devices were exploitable. Not surprising really, since if no one knows they are there, then they won't be included in patching and routine maintenance. This is common amongst such devices.

What Went Right?

There was a competent and persistent CHECK tester who noticed an unknown device responding on an unusual port on the network which did not appear on the diagrams. Investigation revealed the identity of this device and its weaknesses. Having established the potential for exploitation, permission was sought and granted to attempt to execute an aggressive but non-destructive exploit. The results were startling.

All your base are belong to us! [4]

Authentication Bypass

A way to bypass the authentication on the ILO was identified and exploited; a new account was added to the iLO and used to obtain administrative control over the hardware.

• This was possible due to it running unpatched vulnerable code.

Boot custom OS

Access via the ILO administrative interface was used to boot a custom OS on the server hardware. This was facilitated by mounting a remote device via the ILO interface: a bootable “live CD” image was used. During the server reboot process, it was possible to access the boot menu and set it to boot the CD image instead of the main operating system on the server drives. The CD image contained a “live” Linux image, which booted on the server hardware.

Access to Server File System

The Server, now running the custom OS, was able to mount the hard drive and search through files and folders at will. By writing to the hard drive, it was possible to trojan the real OS, bypassing the usual OS controls entirely, including the OS's Virus/Malware detection software.

• Encrypting the OS drive makes this a lot harder

A number of tools were uploaded to the root directory of the main system drive and the “Sticky Key” binary on the system was replaced with the “cmd.exe” executable.

Reboot into Server's own OS and Break In

The Server was rebooted into its normal OS. From the login screen, the “Sticky Key” function was triggered by pressing the SHIFT key 5 times. This resulted in the replacement binary running (CMD.EXE), presenting a command prompt at the login screen running with SYSTEM user privileges.

Subvert User Accounts (including Domain Admin Accounts)

It was found possible to extract user names and plain text passwords from the system[2]. One of the accounts compromised by this process was a member of the Domain Admin group for the Windows Domain.

This means that we (the attacker) now have Domain Administration credentials for the entire Windows Estate – all of the Windows Servers within this particular system. This is a complete and catastrophic failure of security.

• There are many ways this could have been less bad with better configuration.
• A more thorough compromise would only have been possible if this could have been exploited from outside of the local network.

Lessons To Learn

If critical security-affecting components are missed from diagrams, then the consequences can be devastating.

Generally the assessment of risk is done based on the diagrams and other documents, rather than by physically grovelling around on the floor of the data centre following wires.

Assuming that something on the network “does not matter” is a recipe for disaster and can and should be severely career-limiting.

So how confident are YOU that YOUR network diagrams show everything that they should show?

Footnotes

[0] This isn't an insult. These form the backbone of the Army, have voices which range effortlessly and rapidly from audible to deafening, and know how to follow rules with breathtaking bloodymindedness. These value of these qualities should not be under-estimated.

[1] Cough like British Airways. Looks like they may have failed some PCI testing but not have implemented their remediation activities. That could cost them dearly under GDPR.

[2] Artistic licence. It was most likely somewhere else.

[3] For the curious, the Windows Credential Editor tool, uploaded earlier, dumped out a number of clear-text domain users and their passwords. One of the domain accounts that was presented was found to be assigned in the Domain Admin group for the Windows Domain.

[4] From around 1998, but re-surfaces from time to time