2. Splitting the Atom

Career stage: 7 years into my IT career I had worked on several Active Directory deployments and was quite confident in my technical capability to build and manage more complex domains and forests. In this next project I was between jobs and looking for a short-term contract to keep me busy.

?

Challenge: I was asked to help a company that has a unique challenge and needed a plan that would save them from a major impact on their IT systems. They had just been informed their parent company was bankrupt and had to close all its businesses, which included the data center that housed the parent domain in the single forest, multi-domain structure. The customer, Company A, would continue to operate under new ownership and needed its systems to continue to run unaffected.

Except it couldn't, not with the parent Domain (the Forest Root) offline permanently. Eventually this would cause irreparable damage to domain trusts, replication issues, and in future prevent domain and forest upgrades.

The official answer to this problem (in case you ever sit an interview like this) would be to create a new Active Directory Forest, migrate all users, groups, computer accounts, and applications, then continue operating in the new domain. That is, at best, a 1-year project with potentially major disruption to all operations.

?

This customer had just 3 months.

?

I'll use several diagrams to explain the project steps before, during, and after, and explain why we did what we did. Here is what the business structure and Active Directory Forest looked like:

Company A operated all its servers and client computers within its own local network and data centers. It has a direct network connection with the Parent Company to replicate with the Forest Root Domain. Whilst Company B and Company C also operate like this, there is no network connectivity between the child companies. They were isolated from each other, yet they all relied on the connection to the parent domain.

When the Parent Company shuts down, Company A would be left with all their own IT systems running normally, except for the loss of connectivity to the domain controllers for the Forest Root Domain.

During the first interview, I was presented with this problem and asked how I would approach it, and what solutions I could recommend, to ensure they could continue to operate after the parent company turned off the lights in their datacenter. I had only one answer:

Split the forest into parallel universes:

1. We take a backup of the domain controller in the Forest Root Domain.
2. We restore the backup to a local server, but keep it disconnected from the Company A network.
3. Connectivity to the parent company is removed.
4. Connectivity to the newly restored domain controller is enabled.
5. We now have domain controllers for both domains, running on the same network.

With the limited options available to them, they chose to go with my plan, I was hired and started work the next week. 3 months had turned into just 9 weeks to complete the work before the firewall rules were cut off and power was shut down at the parent company’s data center. That’s time to plan, replicate, test, and document the entire procedure.

The customer had to have confidence this was going to work first time as there was no going back. Once we cut the connection to the head office, and the copy of the Forest Root Domain was stood up locally, you couldn't revert the changes. I knew it was feasible, but I had to work on the exact steps required to carry out each of the steps laid out above.

During my testing, I created a physically isolated virtual environment with 4 virtual machines running in it. I stood up two domain controllers for the Forest Root Domain (using their latest backups), and two domain controllers for Child Domain A (again, using backups from the production domain controllers). I now had a replica environment where I could fully test all required steps, including the changes of IP addresses, DNS entries, and the removal of orphaned sites and domains (Child Domain B, and Child Domain C). It looked a bit like this:

After 6 weeks researching the Microsoft documentation, working through all the steps, and documenting the exact procedures, I could confidently recreate the steps required in the virtual test environment. The plan was set and clearly communicated with the customers management team - we got the green light to proceed in week?7, a whole 2 weeks before the deadline!

The operation started at 7pm on a Friday evening when there was less risk to business operations, but we had to be finished by 7am on the Saturday, just 12 hours to complete the work. I knew it would only take 4-5 hours, so we had some extra time if needed.

Everything proceeded exactly to plan, until it didn't. There was an issue that we ran into where just one of the steps I had rehearsed in the test environment didn't behave the same in the production environment, and we were past the point of no return. A long story short, we had to call Microsoft support to get some urgent assistance and they responded fantastically. I ended up speaking with 7 different subject matter experts as each one escalated the case to the next one trying to figure out what could be the cause.

After 7 hours of troubleshooting, we discovered the single root cause: the previous owner of the Forest Root Domain had changed a configuration deep inside of Active Directory schema that granted special permissions directly to his administrator account, removing that permission for everyone else. Once the Microsoft experts found this and fixed it, everything carried on as expected. We completed the work before the 7am deadline and spent the next few hours monitoring any issues.

Upon final review, the customer's Active Directory infrastructure was cleaner than it had ever been - no replication errors in the logs, no orphaned objects, sites, or domain trusts, and only those components they required to continue operating for years to come. The official term for their new design is a "Dedicated Forest Root", as long as they continue to operate both domains well, there is no need for a domain migration.

?

What I Learnt: Active Directory is a robust infrastructure if well maintained; designs should be reviewed regularly to check for orphaned objects such as sites, old domain controllers, and domain trusts. I recommended the company further review the Forest Root Domain configuration to find any other modifications, mismanagement, and potential security improvements.

Even the best laid plans might not work but keep working the problem until you get it fixed. Whilst this project was done under pressure, the approach was well reasoned, and the outcome was better than expected.

?

Join me in the next article as we ask, "Where is the perimeter?".