Troubleshoot Hybrid Azure AD Join Autopilot provisioning 0x80070002 error #Iwork4Dell #HAADJ #IntuneConnector

Requirements and technical background

A properly configured Hybrid Azure AD Join (HAADJ) requires the following at minimum:

• A Hybrid Azure AD joined Autopilot profile.
• A Domain Join device configuration profile configured with the domain and distinguished name of the OU to join.
• Azure AD Connect configured to sync the OU you want to join devices to.
• Service Connection Point (SCP) configuration - This can be configured via Azure AD Connect to configure an entry in AD to allow Domain joined devices to lookup AAD tenant info and begin the process of registering with Azure AD during user logon.
• Installation of the Intune Connector - an agent that runs on a member server which will receive and process Hybrid Domain join requests during Autopilot provisioning.

During provisioning there are two separate phases of Hybrid AAD Join which can be correlated with events from the Enrollment Status Page:

• Device Preparation- This is the initial phase before ESP is shown where the service requests offline domain joins to be processed by the server with the Intune Connector installed. The Intune connector agent receives the AD offline join request, creates the computer object in the domain/OU configured in the Domain Join device configuration profile, and then sends the offline blob through the Azure/Intune service and down to the device being Autopilot provisioned.
• Account Setup - Whether User ESP is configured or not, when a user logs on after the Device Setup phase the device being provisioned makes a set of communications between the local Active Directory and Azure Active Directory, with dependencies on the Azure AD Connect service to complete the AAD device registration process. When User ESP is enabled, the user will have to wait until these processes finish, otherwise they'll arrive at the desktop early but any access to Azure services (Company Portal, Teams, etc.) won't work until that's complete, they may even get a message that something's wrong with their Azure account. There are a variety of reasons why this phase gets delayed and can cause errors. A good video on this phase was published by Microsoft, skip ahead to the 1:20 mark for the technical overview.

That video was published about a year after Michael Niehaus's OOFHours blog post where he summarizes both the challenges and delays during user logon, and offers some workarounds:

Identifying issues with the Intune Connector

During Autopilot provisioning the client device will reach a point where it's waiting for the Offline Domain Join (ODJ) blob to be created by the Intune connector which should get downloaded locally to be processed. Whenever there's an issue on the backend, you'll usually see an error message about not being able to join Active Directory after Autopilot eventually times out and fails.

First step is confirming the error message by running Get-AutopilotDiagnostics on the device itself, or against diagnostics logs. The bottom of the script output will confirm ODJ timed out, but you should also see something like this at the top of the output:

The highlighted EntDMID is explained on Microsoft's documentation as two things:

1. A dummy policy which is tracked by ESP.
2. And as: "The character string that contains the unique enterprise device ID. The value is set by the management server during the enrollment process by way of the DMClient configuration service provider". In other words, this is the Intune Device ID which is recorded in the registry that the diagnostics script reports from, and something we can confirm by pulling up the Autopilot registration record and clicking on the Intune device record (which also means you don't need to run the diagnostics script if you know the device serial number and have access to Intune):

As previously mentioned, you need to use Event Viewer to search for issues with the Intune Connector, but now you know what you can look for by searching for both logs with this ID.

You can do this by logging on to the server with the Intune Connector agent installed, opening a PowerShell window and typing the following (after adjusting the EntDMID to one from your device) to get a quick status of the join process for a single device.

# CHANGE THE LINE BELOW WITH YOUR DEVICE ID

$EntDMID = "b2f80338-1193-474b-9b82-4266f239de20"

cls; Get-WinEvent -LogName Microsoft-Intune-ODJConnectorService/Operational, Microsoft-Intune-ODJConnectorService/Admin | ? { $_.message -match $EntDMID } | select timecreated, id, leveldisplayname, containerlog, message | fl

This short script queries both sets of ODJ event logs for the Device ID within the message which is much faster and more flexible than dealing with the Event Viewer manually.

As a demo, I purposefully created an invalid Device name prefix ... which is kind of surprising that Microsoft doesn't do input validation against no? Either way you can see the EntDMID is the Intune Device ID and the command returns entries from both Admin and Operational logs in a single view. Check out what I configured as a device prefix.

A "normal" Offline Domain join should show the three expected events for Download, Create, Upload:

This is just an example; you can adjust the script further to include other error messages or gaps such as when it can't connect to Microsoft services or include multiple devices in an array.

What could cause errors with the ODJ Connector?

As mentioned, there's no input validation when creating an Offline Domain join configuration policy so you have to verify the domain and OU are correct, and Delegation of permissions is also correct. On occasion you'll see issues when the Intune Connector isn't able to connect to the backend server due to transient issues with the Microsoft service. Other times you may see issues where the ODJ Blob upload had an error. In those cases, you'll need to do some additional troubleshooting which includes talking to both your network and security teams for any polices that would block, throttle, or otherwise increase latency with normal communication. I've worked with customers where the connector seems fine when first booted, but once 5-10 ODJ requests get processed all the uploads become limited. In each of these cases it was always a policy from a security agent running on the server that needed to be adjusted.

When ODJ Blob uploads fail it seems like the upload queue becomes saturated, but I can't say for sure because the Intune Connector technical details aren't documented by Microsoft, but this is a sample of what you may see in the logs, notice the retry count:

I've seen logs with retry counts in the thousands for a single device. A reboot of the server can usually take care of that temporarily, and if you've fixed the issue causing the communication problem it should resolve itself, however you should always contact Microsoft Support just in case.

Update March 2024: Abhishek Yadav recently documented his problem was related to the Intune Connector service not starting up after reboot.

Other things to look out for

Everything above is for a basic Offline Domain Join scenario and only focused on the Intune connector issues with a separate brief on the dependent processes during user logon. There are a few scenarios like multi-domain forest and multiple connectors that will increase the complexity of configuration and troubleshooting.

As far as the user logon phase, if you need to support ADFS or single-sign on products that may require additional configuration to support WS-Fed and WS-Trust protocol support (each vendor has their own KB for this). Remote users will need a VPN product that allows users to start and connect to VPN from the lock screen to help complete AD logon authentication. Security products may need to be tuned to not interfere with protocol communications.

Workarounds that may help the user logon phase include:

• Disabling User ESP - to prevent timeouts waiting for all the background processes to complete. It's not something explicitly documented by Microsoft, but their support staff will always mention it as a first step.
• Running a script on the server where AAD Connect is installed that can monitor and kick off Azure AD Connect delta sync when computer objects with User Certificate attributes are found.
• Running a script on the Hybrid AAD Join client device which monitors and informs users of the Device Registration status until it's successfully completed.

Michael Niehaus's original troubleshooting post mentions "skip connectivity check" being required to be enabled, but it seems Microsoft has made a change to Windows where this isn't required anymore as I was able to pre-provision Windows 11 22H2 with "skip connectivity check" as disabled off-premises, and hybrid join completed successfully.

Given all the complexity and risk for errors some may argue that not using HAADJ at all is the best solution, but that's not possible for every company for a variety of reasons.

Will Microsoft ever improve Hybrid Azure AD Join?

There's a great episode from the excellent Unpacking Endpoint series where the panel debated the pros and cons of HAADJ, its challenges, and taking in questions from viewers on the topic. At the 52:27 mark, Jason Sandys, Product Architect on the Intune and Config Manager team said in reference to the pain of device naming limitations and practices with HAADJ:

Doing Hybrid Azure AD domain join with Autopilot is kind of painful. [...] It's not the complete solution, it was never intended to be, and we have no intent of working on it anymore.

In a word: no. We're stuck with whatever we have for HAADJ possibly forever.

Hopefully, this article helps you make the best of your situation with Hybrid join, but you should really consider making the investment to migrate towards Azure AD Join instead.

Of course, if you ever need help our delivery teams deal with everything from brand new Intune setup to migrating or expanding existing infrastructure, including Co-Management, GPO Migrations, factory services including Autopilot registration with pre-provisioning, troubleshooting, comprehensive cybersecurity services and more: contact Dell Services or your Dell sales rep for more info, or leave a comment below.