Applying SCCM 2403 Hotfix KB28290310

Generally applying SCCM\MECM In-console hotfixes is a straight forward process, with not much hassle. (If done properly!!!)

You have to follow similar steps like applying an in-console update, backups, etc. (I'll mention them in detail in a dedicated article on Applying SCCM In-Console Update).

But, this hotfix KB28290310 is a little different than the others, it's responsible to upgrade your CMG VM Scale set VM OS to build version 20348.2402

There is not much to do on our end apart from Installing the hotfix, as the VM is managed by Microsoft, and it will be upgraded in background by Azure PaaS team. You just have to make sure there is no blocker from your internal infrastructure for this upgrade.

Its recommended that you don't modify the VM settings on your own as it can hamper the configuration, like Disk type, encryption type etc.

Now there are some issues or blockers which can stop the VM OS from being upgraded, and these our your Internal Azure Infra configurations. I'll mention a few here but there can be more conflicts which can block the VM upgrade, you can find them in the Activity Logs.

Let's get in to the dirty details:

So assuming you have already installed the Hotfix KB28290310 in the SCCM console.

Now look at the CloudMgr.log it will show the initialization of the Deployment task:

At this point Its necessary that the MP role is active, sometimes after the installation MP role becomes unresponsive due to IIS issues and SCCM Client installation issue on the Site server.

To check this make sure MPControl.log is showing Status code 200, test: OK, if you have any other issues or Status Code 500 resolve them first, 2 of the most notable issues are:

1. MP role waiting for SCCM Client to install first as it shows as one of the 3 pre-requisites.

This issue creates a loop, SCCM Client couldn't install because the MP is not active, and the MP couldn't install because it needs the client to be installed first!

The resolution is to run ccmclean.exe and remove the client, which removes it as a pre-req as well.

Make sure the other 2 pre-reqs are installed SQLNativeClient and MSODBC

2. Another issue is Status code 500, which generally points to IIS related issues, check the IIS logs to find the exact cause and resolve it.

recently its been the 64 bit and 32 bit issue. (Another post on this later on).

Assuming the MP is running fine, lets get back to CMG logs, or lets move to Azure Activity logs instead:

Login to your azure portal> resource group (Where the CMG component is installed)> Select the VM Scale Set> Activity Logs

If Every thing went fine and there were no issues and conflicts you'll see the below entry:

What happens inside?

1. It gathers information about the VMSS (Microsoft.Compute resource) like resourceGroupName, resourceProviderName, resourceType, resourceId, status, subStatus, subscriptionID, tenantID etc.
2. Next It checks if there are any Internal Compliance Policies, If yes it will check the audit level (this is the place where you can face failures, if any of the compliance checks doesn't apply on the VMSS, fir example "Encryption at disk")

If this part succeeds you'll see events like these, depending upon how many policies are applied:

If it fails, you'll see the below for each policy that is not met:

To see what caused the failure, click on the event and go to JSON view, scroll to the end to see the exact reason of failure:

In the below scenario it failed because the compliance check wants:

Virtual machines and virtual machine scale sets should have encryption at host enabled        

This is because The VM type offered in CMG VMSS (except the Lab B2s size) are using Ultra Disks which do not support "Encryption at disk" it relies on TLS 1.2 encryption instead. This is something which MS should look at.

Well you need to look at all the failures reported here and apply the appropriate solution, generally in these case the solution is to temporarily change compliance level to informational and once the task is complete change it back to Restrict.

OK, assuming all the policy conflicts are resolved, The next message you'll see in activity logs is "health Event Activated" which is pre task for starting the platform upgrade:

If you click on this event and explore in JSON view

The background Upgrade task puts the resource in Downtime mode:

The main Event which runs the upgrade is "Create and Update Virtual Machine Scale Set" if you expand this event you see the sub events:

• Get Virtual Machine Scale Set - This captures\retains the properties of the VMSS probably to restore after the upgrade.
• 'auditIfNotExists' Policy action & 'audit' Policy action. - To recheck all the compliance\Security policies.
• Create or Update Virtual Machine Scale Set - Verify all the policies, start and finish the OS Upgrade task.

Once these events are completed with success several "Health Events" will run and check the availability of the VM and mark it as "Updated".

You can see this whole process in the CloudMgr.log as well:

Search for "CreateVMSS" This is where the upgrade tasks begins

If there is any issue or error, it will be logged as:

The it logs the DSC task:

The below entries show the successful completion of the upgrade:

How to verify if the update completed successfully?

Check the version of VM Scale Set OS Build in Azure> Resource Group> VM Scale Set> Settings> Operating System It should be Version: 20348.2402.XXXXX

Make sure the services are running properly, no error in CMG logs, Azure Activity logs, or CMG Connection Analyzer etc.

This article is a small part of SCCM 2403 Upgrade, which is coming shortly.

Keep digging logs!!!! ???????????