Fix Apache log4j2 in vCenter Appliance

There are two detailed articles from VMware outlining the steps necessary within the VMware vCenter appliance for modifying the files to insulate your vCenter appliance from the Apache log4j2 vulnerability. However, VMware leaves out a few pre-requisites and interim steps that you might need to know about to make this task easier.

First, let's start with the prereqs. You'll need to download the python scripts attached to the two VMware articles that outline the remediation steps (links provided below). Then, you're going to need Putty or some other SSH client to be able to log in to the vCenter appliance and run these scripts. Thirdly, you'll need WinSCP or a similar file transfer client to move the files from your PC to the vCenter appliance.

Once you have the three python script files downloaded (vmsa-2021-0028-kb87081.py, remove_log4j_class.py and vc_log4j_mitigator.py), launch your web browser and navigate to the admin interface for your vCenter appliance (https://vcenterdnsname:5480), and log in as root. This is not the root user for your ESX host, it's the root user for the vCenter appliance. If you can't log in, your root password may have expired since for some idiotic reason VMware thought it was a good idea to set the default for root's password to automatically expire in 90 days, after which you're permanently locked out and there is no option to add another root-equivalent user. Anyway, I've provided a link to an article below on how to reset the root password of a vCenter appliance, should that be necessary. Once you've logged in as root, navigate on the lefthand menu to the Access submenu, then go to the upper right side and click EDIT, to enable the BASH SHELL. Set the timeout to 30 minutes or longer. Log out of the admin interface.

Next, navigate to the main menu of the vCenter server (https://vcenterdnsname) and log in using either [email protected] or an Active Directory admin user authorized for vCenter. Find your vCenter appliance in the VMs list, right-click on the VM, go to Snapshots > Take a Snapshot > OK and then wait for that to finish (the user interface may pause for a moment while this happens). Verify that the snapshot has been completed, and then log out.

Now, we're going to transfer the python scripts to the vCenter server. I'll use the WinSCP example here. Since the default shell environment for vCenter is the appliancesh shell, and not bash shell, you'll have to do one of two things. Either SSH to the vCenter appliance and change the default to BASH shell to be able to use secure copy (SCP), or simply use SFTP which can be accessed when the appliancesh is the default shell environment.

In WinSCP new session, select SFTP as the file protocol, type in the host IP address or DNS name of the vCenter appliance, leave port as 22, and the username will be root and type in your root password, then click advanced. See the screenshot above for the protocol options to replace "Default" in the SFTP server block, with shell /usr/libexec/sftp-server, then click OK to close that windows and click LOGIN. Navigate in the righthand pane of the WinSCP client to the ./tmp folder and copy the two python scripts to this folder. You can then end your WinSCP session.

Launch your putty client to connect to the vCenter server via SSH, port 22. If you need to document session input and output for audit reasons, make sure to enable logging in the putty client. Once connected, log in as root. You'll be presented a Command> prompt. Type in shell. You'll then be given a root@vcsaXX [ ~ ]# prompt (with the version of the vCenter server replacing the XX. Type in cd /tmp and press enter. You can then type ls -l *.py to verify that the three python scripts are in this directory.

Now, let's pause here to consider the potential impact of the next two actions. When you run the python scripts several services are STOPPED on the vCenter appliance. This could impact any service that depends on vCenter, such as VMware replication, VMware Backup, Zerto Replication, Veeam Replication, Veeam CDP, Veeam Backup, and other such programs as well as anyone attempting to use the GUI interface of vCenter. Before you run the python scripts you'll need to notify users of the outage and potentially pause other programs to reduce the possibility of an interruption of services for replication or backups. The two article links below outline the services that will be stopped by the python scripts. Review those to determine any potential impact before running the two scripts.

Now that you've considered the consequences of these next steps and taken actions to reduce any potential impact, you can run the first script by typing: python vmsa-2021-0028-kb87081.py from the root@vcsaXX [ /tmp ]# prompt and you'll be then prompted by the script to answer Yes or No as to whether you want to stop services and run the script. Once you answer YES, the script will stop the appropriate services, back up the files that will be modified, and then apply the modification(s) to the necessary files before restarting the associated services. Depending on the speed of your ESX host and the underlying disk drives this process can take 20 minutes or longer. Be patient. You'll then see output similar to this:

Once the first script completes, we can now run the second script by typing: python remove_log4j_class.py and hitting enter. You will be prompted to answer yes or no to continue. Once you type y and hit enter, the script will run with screen output showing files being backed up before modification, and then you will be returned to a command prompt with a message containing the date and INFO main: Done. You can verify that the script worked successfully by typing: python remove_log4j_class.py -r which should then echo that the list of vulnerable files is now empty. Script three (added 12/21/2021) named vc_log4j_mitigator can be run with the command python vc_log4j_mitigator.py -r with the -r indicating running the script in "dry run" mode that checks to ensure all changes are made, without actually modifying any of the files. NOTE: If you are running vCenter Server Appliance 6.0U3j, the remove_log4j_class scripts may not work. You will have to manually remediate using the steps outlined in the 87081 knowledgebase article.

Once you verify that your vCenter is working properly and that all dependent services are also functioning as they should, you can then delete the VM snapshot that we created earlier. This is important due to the fact that a long-running snapshot file can become rather large and not only slow down your vCenter server but present many other problems later such as exhausted disk space.

Here are the links to the two VMware KB articles. Each has the python script(s) attached on the right-hand side of the screen in the block labeled Attachments: https://kb.vmware.com/s/article/87088 https://kb.vmware.com/s/article/87081#vCenter67

Here's a link to the VMSA-2021-0028 Advisory that is continuously updated: https://www.vmware.com/security/advisories/VMSA-2021-0028.html

If you want to use SCP instead of SFTP to copy files to your vCenter appliance, here are the instructions on changing the default shell from appliancesh to bash. Just remember that once you're done copying the files via SCP, you'll need to change the default shell BACK to appliancesh. https://kb.vmware.com/s/article/2100508

And, last but not least, here's the link to the article on how to reset the root password of your vCenter appliance if it expires. Make sure that IF you have to do this you then log back into the vCenter appliance admin console (port :5480), go to the Administration menu, go to Password expiration settings, click EDIT, and set Password Expires to NO. https://kb.vmware.com/s/article/2147144

*note1: Tuesday 12/21/2021 - Added vc_log4j_mitigator.py script reference due to it being added to VMware KB 87081.