Part 1.12 - Migration performance and best practices

In the last section, we talked about throttling and how it applies to migrations.

This section deals more with factors that can cause performance issues. Although throttling can cause performance issues, since it was extensively covered in the last section it will not be covered again here.

Official documentation for migration performance and best practices can be found in the following documents:

Microsoft 365 and Office 365 email migration performance and best practices Performance factors and best practices for hybrid migrations

Migration Speed

IMPORTANT: Referencing the above documentation links, Microsoft no longer publishes minimum expected speeds for migrations. Typical throughput will usually range between 0.3GB/hr to 1.0GB/hr (or better), but this is not guaranteed. That being said, if the source environment is fully optimized with no performance issues and very low latency is observed we have historically seen many mailboxes migrate with speeds in excess of 3+GB/hr. On the other hand, when source environments have problems you can see migration speed drop astronomically (<.1GB/hr). For best practices and best overall throughput, avoid migrating individual mailboxes. Instead, migrate batches of 20 or more mailboxes. Historical average migration throughput was calculated based on 20 or more concurrency.

Migration Performance

When we talk about migration performance, there are several things to keep in mind.

1. Health of the servers (both source and target)
2. Configuration settings to optimize migrations (both source and target).
3. Network speed and latency. This includes both the local network as well as available Internet bandwidth. Settings to optimize Exchange for latent connections will be covered under the above Configuration Settings.
4. Number of items to migrate (per mailbox)
5. Other factors

1. Server health

Server Health and the throttling mechanism associated with resource health was discussed fairly extensively in the last module, so just understand that this has the potential to be a big cause of delays. Please refer to the previous module for how to view throttling data in the move report.

2. Configuration settings

To determine whether you might benefit from changing configuration settings, you need to look at several things.

Exchange Online Config settings

First, Exchange Online provides the following default configuration settings, which can be viewed by running the command Get-MigrationConfig | fl

MaxNumberOfBatches      : 100
MaxConcurrentMigrations : 300
Features                : MultiBatch, PAW
        

The most important number here is the MaxConcurrentMigrations. This is the maximum number of migrations that can concurrently be active between all migration batches. It also defines the maximum setting that can be configured on a Migration Endpoint.

These are the default settings for a Migration Endpoint. Below is an example of an Exchange Remote endpoint type (MRS), which is used for Hybrid migrations.

EndpointType                           : ExchangeRemoteMove
MaxConcurrentMigrations                : 20
MaxConcurrentIncrementalSyncs          : 10
        

MaxConcurrentMigrations = the maximum number of migrations that can be actively synchronizing data for all batches configured to use this migration endpoint. MaxConcurrentIncrementalSyncs = the maximum number of incremental syncs for mailboxes that are in the Synced state (95%). This is also used when completing migrations, as part of completing a migration performs a Final Incremental sync. Note that the MaxConcurrentMigrations is only set to 20 by default and MaxConcurrentIncrementalSyncs is only set to 10 by default, but both these values can easily be changed using Set-MigrationEndpoint.

IMPORTANT: Please ensure that Admins do NOT set MaxConcurrentIncrementalSyncs to 0. This will prevent migrations from completing.

If there are multiple Migration Endpoints configured, the MaxConcurrentMigrations values configured on each endpoint cannnot exceed the MaxConcurrentMigrations value defined in the Migration Config.

NOTE: The MaxConcurrentMigrations value in the Org Config applies separately to each type of migration endpoint. For example, you can have an ExchangeRemoteMove endpoint with MaxConcurrentMigrations set to 300, and you can also have an IMAP endpoint with MaxConcurrentMigrations set to 300.

Now, lets get to how these config settings can impact migrations.

Migration Config

If there are many on-premises Exchange servers hosting mailboxes where multiple Migration Endpoints are being used, it may be beneficial to increase MaxConcurrentMigrations. This can be done by running the diagnostic "Adjusts the migration configuration MaxConcurrentMigrations value for a tenant" The maximum value that the diagnostic will increase to is 1000. Although MaxConcurrentMigrations can still be increased with a single Migration Endpoint (example: multiple MRS Proxy servers behind a Load Balancer), the benefit of increasing concurrency decreases as the potential for flooding their network with connections increases which could then slow down all migrations. If a admin insists that MaxConcurrentMigrations be increased and they will only be using a single Migration Endpoint, please make sure they understand the potential impact to their network. If a admin has multiple MRS Proxy servers, instead ask them if they can publish each MRS Proxy server individually, then create a separate Migration Endpoint for each MRS Proxy server. Under no circumstances should MaxConcurrentMigrations be increased if there is only one migration endpoint with a single MRS Proxy server. If concurrency is increased for a single MRS Proxy server, the default MRS configuration settings will cause migrations to be stalled due to reaching the max MRS connection limit.

Source Exchange config settings

Maximum performance can be achieved by ensuring the following:

• Migrate mailboxes from servers closer to Office 365 datacenters. What we mean here is that admins that use a Hub/Spoke model where regional satellite offices access the Internet via a main Hub site are by nature going to encounter greater network latency. To provide the best overall experience with the lowest possible latency, we recommend that each location with mailboxes have Internet access and have one or more MRS Proxy servers published that is connected to a Migration Endpoint. Additionally, admin should ensure that the DNS servers used by Internet-connected servers are local DNS servers. Although migrations are Pull migrations (connection originates from Exchange Online), using non-local DNS servers could still cause the responses to be routed in a non-optimal way. Consider using Tracert to check the network route that requests are taking. ?? NOTE: Tracert utilizes ICMP packets to test the hops to the target IP. Some routers block ICMP, so using Tracert may result in some requests timing out. This is normal and is not an issue.Example:

PS C:\> tracert outlook.office365.com

Tracing route to DSM-efz.ms-acdc.office.com [52.96.7.226]
over a maximum of 30 hops:


1     4 ms     1 ms     2 ms  gateway.home [192.168.254.254]
2    51 ms    23 ms     3 ms  47.185.0.1
3     7 ms    10 ms    14 ms  te-1-10-0-20---0.lcr01.irng.tx.frontiernet.net [172.102.54.66]
4     6 ms     6 ms    12 ms  ae7---0.scr01.dlls.tx.frontiernet.net [74.40.3.17]
5    10 ms     8 ms     8 ms  ae0---0.cbr05.dlls.tx.frontiernet.net [45.52.201.121]
6    11 ms     7 ms     5 ms  frontiercommunicationsofamericafca.ier02.dfw32.ntwk.msn.net [104.44.44.199]
7    27 ms    43 ms    17 ms  ae25-0.icr04.dsm05.ntwk.msn.net [104.44.230.130]
8     *        *        *     Request timed out.
9     *        *        *     Request timed out.
10     *        *        *     Request timed out.
11     *        *        *     Request timed out.
12     *        *        *     Request timed out.
13    23 ms    20 ms    20 ms  52.96.7.226        

When analyzing the Tracert output, ensure that there are not excessive hops.

• Network Speed and latency. Check Internet latency as well as latency observed during moves. High round trip latency between On-Premises Exchange servers and Exchange Online(>1000ms) will slow down migrations. Using the sample command from the previous item, Tracert can be used to display observed one-way latency to Exchange Online servers. To get a better idea of actual round trip latency, use the move report. During migrations, MRS performs periodic "dummy" network calls to the on-premises MRS Proxy endpoint in order to measure round-trip latency. No data is transferred during these calls - they are just injected to calculate round-trip response time from the MRS endpoint. This information is visible in the move report.

Get-MoveRequestStatistics "move identity" -IncludeReport -DiagnosticInfo "Verbose, ShowTimeSlots, ShowTimeLine" | Export-CliXml movereport.xml        

The latency information is available in the Session Statistics section of the report. For Onboarding moves it will be displayed as SourceLatencyInfo. For Offboarding moves it will be displayed as DestinationLatencyInfo

$movereport = Import-CliXml movereport.xml $movereport.Report.SessionStatistics | fl *latency*        

Within the latency output, you will see several values

Current = most recent sampling call

Avg = Average of all latency sampling calls

NumberOfLatencySamplingCalls = The total number of network sampling calls made

Min = the lowest observed latency of all sampling calls .

Max = the highest observed latency of all sampling calls

It is worth noting is that because no data is transferred during these latency sampling calls, we can't really calculate the actual latency associated with migrating data. These calls are only intended to give you an overall idea of network latency.

For highly latent connections, consider implementing the following values in the on-premises MRS config:

Exchange 2013 and higher

Find the MRS config file under (Default location) : C:\Program Files\Microsoft\Exchange Server\V15\Bin\MSExchangeMailboxReplication.exe.config

<MRSConfiguration
  ....existing content
  ExportBufferSizeKB="512"
  ExportBufferSizeOverrideKB="7500" <---add this line
  OldItemAge = "366.00:00:00"
  ....existing content
  />        

IMPORTANT: When adding ExportBufferSizeOverrideKB, the DataImportTimeout setting also needs to be adjusted.

For Exchange 2013 and higher, this setting is also located in the MSExchangeMailboxReplication.exe.config file.

For Exchange 2013 this setting is adjusted under the MRSProxyConfiguration section of the MSExchangeMailboxReplication.exe.config file.

For Exchange 2016 and higher this setting is adjusted under the MRSConfiguration section of the MSExchangeMailboxReplication.exe.config file as the MRSProxyConfiguration section has been deprecated.

Example: 7.5MB / (60 seconds) = 131,072 bytes/second (for reference 1024kb/s = 1 Megabit/s)

This means you can never drop below this rate. You probably want to actually artificially inflate DataImportTimeout a bit more than the computed minimum. Below is an example configuration increasing from default of 1 minute to 5 minutes. The highest recommended value is 20 minutes.

Exchange 2013

<MRSProxyConfiguration
  MaxMRSConnections="100"
  DataImportTimeout="00:05:00"  <---add this line or configure this setting if already present
  />        

Exchange 2016 and higher

<MRSConfiguration
  ....existing content
  MaxMRSConnections="100"
  DataImportTimeout="00:05:00" <---add this line or configure this setting if already present
  />        

In addition to the above mentioned settings, if there are dedicated MRS Proxy servers that are used solely as MRS Proxy endpoints and are not hosting mailboxes then admins can also consider increasing the Max MRS connections. When migrating many mailboxes, it is possible that

IMPORTANTAfter making changes to the config file, the Mailbox Replication Service must be restarted for the changes to be effective.

To verify the settings that are being used, Get-ExchangeDiagnosticInfo can be used to show the config settings that are loaded into memory.

[xml]$diag = Get-ExchangeDiagnosticInfo -Server <server name> -Process MSExchangeMailboxReplication -Argument config
$diag.Diagnostics.Components.config.AppConfigDriver.MRSConfiguration |fl DataImportTimeout, ExportBufferSizeOverrideKB

DataImportTimeout          : PT5M
ExportBufferSizeOverrideKB : 7500        

• Network speed and latency

There isn't a lot that we can troubleshoot under this topic, but it is important for admins to know how much available network bandwidth exists. A 1GB Internet connection doesn't do any good for migrations if it is 99% utilized all the time. Internet Routers may have functionality that shows how much of the connection is utilized. If the routed does not have that functionality, other software may be available. Testing should be done to determine available bandwidth both before and during migrations so as to ensure migrations do not impact overall network stability. Some admins may have traffic shaping network devices that can allocate a specific amount of bandwidth to certain processes. Make sure that you ask admins if they have implemented any network devices such as this, as they have the ability to throttle migrations from a network perspective.

• Number of items to migrate

Check item count with Get-MailboxStatistics and make sure that admins understand that mailboxes with high item counts are by nature going to migrate more slowly. Also check Mailbox Folder Statistics using Get-MailboxFolderStatistics to ensure there are not empty folders. All things considered, if two mailboxes are the same size the mailbox with fewer (larger) items will migrate faster. This is because MRS has to make fewer calls to migrate the same amount of data.

• Other factors

When possible, if a large number of mailboxes need to be migrated admins should bypass Load Balancers and other network devices such as Reverse Proxies, Outbound Proxies and Network optimizers. Although many load balancers and other network devices have settings that are optimized for Exchange, they add a layer of complexity that can make troubleshooting difficult. Admins should be prepared that when troubleshooting migrations, we may ask them to bypass such network devices. If bypassing said devices resolves the problem (performance or otherwise), they will be asked to contact the vendor of the device for further troubleshooting.