Troubleshooting migration issues in Exchange hybrid environment

If you are having issues determining what the best Migration Approach is for your environment please visit the Exchange Deployment Assistant.

NOTE: This troubleshooter will not help you with troubleshooting Staged, Cutover, or IMAP migrations.

Were you able to initiate the mailbox move?

We need to determine if the mailbox move was successfully initiated, which means you were able to either go through the Exchange Administration Center (EAC), Exchange Management Console (EMC), or Remote PowerShell to begin the move request and you had no issues getting the request started.

We need to determine if the mailbox move was successfully initiated, which means you were able to either go through the Exchange Administration Center (EAC), Exchange Management Console (EMC), or Remote PowerShell to begin the move request and you had no issues getting the request started.

Mailbox moves are more likely to succeed when they are initiated from the EAC in Exchange Online. Please connect to the EAC in Exchange Online and see if you can initiate the move from there.

Remove migration endpoint

1. Log into https://portal.MicrosoftOnline.com with your tenant administrator credentials.
2. In the top ribbon, click Admin and then select Exchange.
3. Click Migration.
4. Click on the ellipses (...) and select Migration endpoints.
5. Select the endpoint that is listed as Exchange remote move.
6. Click on the trash can to delete the endpoint.

On-Boarding Steps

1. Log into https://portal.MicrosoftOnline.com with your tenant administrator credentials.
2. In the top ribbon, click Admin and then select Exchange.
3. Click Migration > click + and then select select Migrate to Exchange Online.
4. On the Select a migration type page, select Remote move migration as the migration type for a hybrid mailbox move.
5. On the Select the users page, select the mailboxes you want to move to the cloud.
6. On the Enter on-premises account credentials page.
   Important: provide your on-premises administrator credentials in the domain\user format.
7. On the Confirm the migration endpoint page, ensure that the on-premises endpoint shown is the CAS with MRS Proxy enabled.
8. Enter a name for the migration batch and initiate the move.

Off-Boarding Steps

1. Log into https://portal.MicrosoftOnline.com with your tenant administrator credentials
2. In the top ribbon, click Admin and then select Exchange.
3. Click Migration > click + and then select the select Migrate From Exchange Online.
4. On the Select a migration type page, select Remote move migration as the migration type for a hybrid mailbox move.
5. On the Select the users page, select the mailboxes you want to move to the cloud.
6. On the Enter on-premises account credentials page.
7. Enter the on-premises Database Name, this can be retrieve by running Get-MailboxDatabase from EMS.
   Important: provide your on-premises administrator credentials in the domain\user format.
8. On the Confirm the migration endpoint page, ensure that the on-premises endpoint shown is the CAS with MRS Proxy enabled.
9. Enter a name for the migration batch and initiate the move.

When you are moving a mailbox to or from the cloud we make a connection to the on-premises environment to the MRSProxy endpoint. Please verify that the MRSProxy endpoint and the WSSecurity authentication type are enabled.

1. Open the Exchange Management Shell on the Exchange 2010 or 2013 hybrid server.
2. Check to see if the MRSProxyEnabled and WSSecurityAuthentication are both set to True. To do this, run the following cmdlet, The word Server in the below cmdlets should reflect the names of the external facing Exchange servers:
   Get-WebServicesVirtualDirectory -Identity "Server\EWS (default Web site)" |fl Server,MRSProxyEnabled,WSSecurityAuthentication
3. If either is false run the following to enable the MRSProxy and set the authentication required to perform the move. To do this, run the following cmdlet:
   Set-WebServicesVirtualDirectory -Identity "Server\EWS (default Web site)" –MRSProxyEnabled $true – WSSecurityAuthentication $True

Note: These settings should be configured on all of the external facing Exchange servers.

You need to ensure that you have your firewall configured to allow certain EWS and Autodiscover endpoints to come through to the Exchange servers without being authenticated at a perimeter device. Additionally, you need to ensure that the migration requests are not treated like a denial of service attack.

Firewall endpoint/pre-authentication settings

The following are the instructions for how to properly publish EWS and Autodiscover via TMG, but you can apply this logic to your own device. This link will provide explicit steps for TMG, but at a high level you need to do the following:

1. Create a new publishing rule (often using the same listener that is already in place) that does not require pre-authentication.
2. Ensure that the rule applies to any traffic that comes over the following paths.
   • /ews/mrsproxy.svc
   • /ews/exchange.asmx/wssecurity
   • /autodiscover/autodiscover.svc/wssecurity
   • /autodiscover/autodiscover.svc
3. Ensure that this new rule is higher in priority than any existing Exchange Related Firewall rules.

IDS settings

Hybrid Migrations can sometimes be treated like a denial of service attack by certain devices. The following logic can be applied to any intrusion detection system, but it was written for TMG specifically.

1. Open the Forefront TMG management console, and then in the tree click Intrusion Prevention System.
2. Click the Behavioral Intrusion Detection tab, and then click Configure Flood Mitigation Settings.Collapse this imageExpand this image.
3. In the Flood Mitigation dialog box, follow these steps:
   • Click the IP Exceptions tab, and then type the IP addresses that the Office 365 environment uses to connect during the mailbox move operation. To view a list of the IP address ranges and URLs that are used by Exchange Online in Office 365, visit the following Microsoft website:
     http://help.outlook.com/en-us/140/gg263350.aspx
   • Click the Flood Mitigation tab, and then, next to Maximum HTTP Requests per minute per IP address, click Edit. In the Custom limit box, type a number to increase the limit.
     Note: The custom limit applies to IP addresses that are listed on the IP Exceptions tab. Increase only the custom limit. In the following example screen shot, the custom limit is set to 6,000. Depending on the number of mailboxes that are being moved, this number may not be sufficient. If you still receive the error message, increase the custom limit.

Having a move request (even a successful one) could prevent a mailbox move from taking place. Connect PowerShell to Exchange Online and verify that there is no move request pending for the user in question. If there is a stale move request you will need to remove it. The following steps outline how to determine if there is an existing move request and remove that request if it exists.

1. Connect to Exchange Online through PowerShell (Not via Exchange Management Shell (EMS)).
2. Run the following command: Get-MoveRequest -Identity 'tony@contoso.com'.
3. If there is a move request that is completed or failed run: Remove-MoveRequest -Identity 'tony@contoso.com'.
   

Often when moving a mailbox to Exchange online it will fail because some of the accepted domains are missing in the service. Please verify if all of the email domains assigned to this user are added and verified in the service.

1. Open Exchange Management Shell.
2. Run: (Get-Mailbox Tony).EmailAddresses.
3. Take note of all of the email addresses that follow smtp: and write the domain names down.For instance, if the results include SMTP:tony@contoso.com, smtp:Tony@foo.com you would need to write down Contoso.com and Foo.com
4. Connect to Exchange Online through PowerShell (Not EMS).
5. Run: Get-AcceptedDomain and ensure that the results include the domain(s) noted in step 3 above.
6. If any of the domains are missing you should add and verify the domain in the portal. Alternatively, you can license the user before you move the mailbox. Usually we use the option of licensing a user when one of the domains stamped on the mailbox is a .local or non-routable domain. Non-routable addresses cannot be added to the service therefore they will not be stamped on the user in Exchange Online.
   

In order for IIS to properly respond to a migration request we need to ensure that the Handler Mappings are in place.Please verify that the EWS and Autodiscover handler mapping are in place.

1. Select the Internet Information Services (IIS) Manager from the Administration Tools menu.
2. Expand the Server name, then Sites, then Default Web Site, then left click on EWS.
3. In the middle pane select the Handler Mappings option.
4. Look to see if there is a mapping with the following:
• Name= svc-Integrated
  
• Path= *svc
• State= Enabled
5. Repeat steps 1 through 4 but this time check the autodiscover virtual directory.
6. If any of the values are missing perform the remediation steps 7 and 8.
   
   
    
   
7. On the Exchange 2010/2013 external facing server(s), open a Command Prompt window, and then move to the following folder:
   C:\Windows\Microsoft.Net\Framework\v3.0\Windows Communication Foundation\
8. Type the following command, and then press Enter:
   ServiceModelReg.exe –r
   

In order for a mailbox move to succeed you need to have a user account in both on-premises and Exchange Online that have a matching mailbox guid. Please verify that the mailbox guid is in place and matches

1. On the On-Premise Hybrid server run the following cmdlet via Exchange Management Shell (EMS).
   Get-RemoteMailbox -Identity “Alias” | fl *guid*
2. Connect Windows PowerShell to the Exchange Online, run the following cmdlet.
3. If there is no mail user in the on-premises environment you can perform the following from EMS:
   Get-Mailbox -Identity “Alias” | fl *guid*
• Create a new user account : New-MailUser -Name Ayla -SamAccountName Ayla -UserPrincipalName Ayla@contoso.com -ExternalEmailAddress Ayla@Contoso.mail.onmicrosoft.com
  
• Ensure you stamp the newly created account with the proper Exchange GUID retrieved from step “2”, this will be done in the On-Premises EMS:
  Set-MailUser Testuser –ExchangeGuid xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Initiating the migration from PowerShell often yields a more actionable error message. The following steps walk you through the process of moving a mailbox from on-premises to Exchange Online via PowerShell

1. Connect to Exchange Online through PowerShell (Not EMS)
2. Then create a variable to store your on-premises admin credentials. The credentials should be stored in the format of contoso\administrator and not administrator@contoso.com.
   $onpremCred = Get-Credential
3. Then run a cmdlet similar to the following, where ‘User’ is the display name for the account you want to move, ‘Webmail.consoto.com’ is the endpoint that has MRSProxy enabled on-premises, and ‘contoso.mail.onmicrosoft.com’ is the routing domain used in Exchange online.
   New-MoveRequest –Identity ‘User’ -Remote -RemoteHostName 'webmail.contoso.com' -RemoteCredential $onpremCred -TargetDeliveryDomain 'contoso.mail.onmicrosoft.com'
   

In order to better direct you in troubleshooting your migration issues we need to determine the current status of the move requests. In order to determine the status please perform the following steps:

1. Connect to Exchange Online through PowerShell (Not via the Exchange Management Shell (EMS))
2. Run the following to check the status of any moves
• Get-MigrationBatch |fl *status*,Identity
• Get-MoveRequest |fl *status*,Identity
  

Mailbox moves and migration batches are not handled at the same priority as client connectivity and mail flow tasks. Therefore if your server or the Microsoft datacenter is under heavy load, the Mailbox moves may be delayed. There is no reason to be alarmed if a move is in a queued state for a good deal of time since the move will more than likely be picked up relatively soon. It is best to not start troubleshooting a stalled move till there had been a long enough delay (such as 8 hours) with no progress or activity.

Note: The following will help you determine if the on-premises server is overtaxed: http://technet.microsoft.com/en-us/library/jj204570.aspx#ExchDataSource

If you are migrating from an Exchange 2003 server it is better for user experience and performance if you move the mailbox first to Exchange 2010 then to Exchange Online.

Some customers choose to do two-hop migrations for large and sensitive Exchange 2003 mailboxes:

• First hop Migrate mailboxes from Exchange 2003 to an Exchange 2010 server, which is usually the hybrid coexistence server.The first hop is an offline move, but it’s usually a very fast migration over a local network.
• Second hop Migrate mailboxes from Exchange 2010 to Office 365.The second hop is an online move, which provides a better user experience and fault tolerance.
  

This section describes best practices for improving network performance during migrations. The discussion is generally because the biggest impact on network performance during migration is related to third-party hardware and Internet service providers (ISPs).The Office 365 Network Analysis Tool is deployed to help analyze network-related issues prior to deploying Office 365 services:

• North America:http://na1-fasttrack.cloudapp.net
• EMEA:http://em1-fasttrack.cloudapp.net
• APAC:http://ap1-fasttrack.cloudapp.net

The first time you click http://na1-fasttrack.cloudapp.net, you will be prompted to install an ActiveX control. Once you install it, you will get a Security Warning as shown below:



Install Java: http://java.com/en/download/index.jsp

Reboot

Go to http://na1-fasttrack.cloudapp.net and you will be prompted to run the application as shown below:

Intrusion detection functionality configured on a network firewall often causes significant network delays and affects migration performance.

Add IP addresses for Microsoft data center servers to your allow list. For more information about the Office 365 IP ranges, see Office 365 URLs and IP address ranges (http://technet.microsoft.com/en-us/library/hh373144.aspx).

IDS settings

Hybrid Migrations can sometimes be treated like a denial of service attack by certain devices. The following logic can be applied to any intrusion detection system, but it was written for TMG specifically.

1. Open the Forefront TMG management console, and then in the tree click Intrusion Prevention System.
2. Click the Behavioral Intrusion Detection tab, and then click Configure Flood Mitigation Settings.Collapse this image
3. Expand this imageIn the Flood Mitigation dialog box, follow these steps:
• Click the IP Exceptions tab, and then type the IP addresses that the Office 365 environment uses to connect during the mailbox move operation. To view a list of the IP address ranges and URLs that are used by Exchange Online in Office 365, visit the following Microsoft website:http://help.outlook.com/en-us/140/gg263350.aspx (http://help.outlook.com/en-us/140/gg263350.aspx).
  
• Click the Flood Mitigation tab, and then, next to Maximum HTTP Requests per minute per IP address, click Edit. In the Custom limit box, type a number to increase the limit.
  Note The custom limit applies to IP addresses that are listed on the IP Exceptions tab. Increase only the custom limit. In the following example screen shot, the custom limit is set to 6,000. Depending on the number of mailboxes that are being moved, this number may not be sufficient. If you still receive the error message, increase the custom limit.

Having a move request (even a successful one) could prevent a mailbox move from taking place. Connect PowerShell to Exchange Online and verify that there is no move request pending for the user in question. If there is a stale move request you will need to remove it. The following steps outline how to determine if there is an existing move request and remove that request if it exists.

1. Connect to Exchange Online through PowerShell (Not via Exchange Management Shell (EMS))
2. Run the following command: Get-MoveRequest -Identity 'tony@contoso.com'
3. If there is a move request that is completed or failed run: Remove-MoveRequest -Identity 'tony@contoso.com'

Mailbox moves are more likely to succeed when they are initiated for the EAC in Exchange Online. Please connect to the EAC in Exchange Online and see if you can initiate the move from there.

Remove migration endpoint

1. Log into https://portal.MicrosoftOnline.com with your tenant administrator credentials.
2. In the top ribbon, click Admin and then select Exchange.
3. Click Migration.
4. Click on the ellipses (...) and select Migration endpoints.
5. Select the endpoint that is listed as Exchange remote move.
6. Click on the trash can to delete the endpoint.

On-Boarding Steps

1. Log into https://portal.MicrosoftOnline.com with your tenant administrator credentials.
2. In the top ribbon, click Admin and then select Exchange.
3. Click Migration > click + and then select Migrate to Exchange Online.
4. On the Select a migration type page, select Remote move migration as the migration type for a hybrid mailbox move.
5. On the Select the users page, select the mailboxes you want to move to the cloud.
6. On the Enter on-premises account credentials page.
   Important: provide your on-premises administrator credentials in the domain\user format.
7. On the Confirm the migration endpoint page, ensure that the on-premises endpoint shown is the CAS with MRS Proxy enabled.
8. Enter a name for the migration batch and initiate the move.

Off-Boarding Steps

1. Log into https://portal.MicrosoftOnline.com with your tenant administrator credentials
2. In the top ribbon, click Admin and then select Exchange
3. Click Migration > click + and then select the select Migrate From Exchange Online.
4. On the Select a migration type page, select Remote move migration as the migration type for a hybrid mailbox move
5. On the Select the users page, select the mailboxes you want to move to the cloud.
6. On the Enter on-premises account credentials page
7. Enter the on-premises Database Name, this can be retrieve by running Get-MailboxDatabase from EMS
   Important: provide your on-premises administrator credentials in the domain\user format.
8. On the Confirm the migration endpoint page, ensure that the on-premises endpoint shown is the CAS with MRS Proxy enabled.
9. Enter a name for the migration batch and initiate the move.
   

You need to ensure that you have your firewall configured to allow certain EWS and Autodiscover endpoints to come through to the Exchange servers without being authenticated at a perimeter device. Additionally, you need to ensure that the migration requests are not treated like a denial of service attack.

Firewall endpoint/pre-authentication settings

The following are the instructions for how to properly publish EWS and Autodiscover via TMG, but you can apply this logic to your own device. This link will provide explicit steps for TMG, but at a high level you need to do the following:

1. Create a new publishing rule (often using the same listener that is already in place) that does not require pre-authentication
2. Ensure that the rule applies to any traffic that comes over the following paths.
   • /ews/mrsproxy.svc
   • /ews/exchange.asmx/wssecurity
   • /autodiscover/autodiscover.svc/wssecurity
   • /autodiscover/autodiscover.svc
3. Ensure that this new rule is higher in priority than any existing Exchange Related Firewall rules

IDS settings

Hybrid Migrations can sometimes be treated like a denial of service attack by certain devices. The following logic can be applied to any intrusion detection system, but it was written for TMG specifically.

1. Open the Forefront TMG management console, and then in the tree click Intrusion Prevention System.
2. Click the Behavioral Intrusion Detection tab, and then click Configure Flood Mitigation Settings.Collapse this image
3. Expand this image.In the Flood Mitigation dialog box, follow these steps:
   • Click the IP Exceptions tab, and then type the IP addresses that the Office 365 environment uses to connect during the mailbox move operation. To view a list of the IP address ranges and URLs that are used by Exchange Online in Office 365, visit the following Microsoft website:http://help.outlook.com/en-us/140/gg263350.aspx(http://help.outlook.com/en-us/140/gg263350.aspx)
   • Click the Flood Mitigation tab, and then, next to Maximum HTTP Requests per minute per IP address, click Edit. In the Custom limit box, type a number to increase the limit. Note: The custom limit applies to IP addresses that are listed on the IP Exceptions tab. Increase only the custom limit. In the following example screen shot, the custom limit is set to 6,000. Depending on the number of mailboxes that are being moved, this number may not be sufficient. If you still receive the error message, increase the custom limit.

In order for IIS to properly respond to a migration request we need to ensure that the Handler Mappings are in place. Please verify that the EWS and Autodiscover handler mapping are in place.

1. Select the Internet Information Services (IIS) Manager from the Administration Tools menu.
2. Expand the Server name, then Sites, then Default Web Site, then left click on EWS.
3. In the middle pane select the Handler Mappings option
4. Look to see if there is a mapping with the following:
• Name= svc-Integrated
  
• Path= *svc
• State= Enabled
5. Repeat steps 1 through 4 but this time check the autodiscover virtual directory
6. If any of the values are missing perform the remediation steps 7 and 8
   
   
    
   
7. On the Exchange 2010/2013 external facing server(s), open a Command Prompt window, and then move to the following folder:
   C:\Windows\Microsoft.Net\Framework\v3.0\Windows Communication Foundation\
8. Type the following command, and then press Enter:
   ServiceModelReg.exe –r

Often migration issue are cause by corrupt items or mailboxes. These issues can often be resolved by moving a mailbox between two different on-premises mailbox databases. The following walks you through the process of moving a user’s mailbox from one database to another, then moving the mailbox to Exchange Online (if this is an off-boarding request, this step will need to be skipped).

Sometimes a migration batch may become stuck at a certain stage of migration such as “Completing”. You may be able to get past this by cleaning up the old move requests.

1. Open PowerShell (Not via EMS) and connect to Exchange Online.
2. Please run the following to ensure that the move request completion was initiated:
   Get-MoveRequest | ? {$_.Status -eq "AutoSuspended"} | Resume-MoveRequest
3. After giving time for the resumed move requests to complete, run the following:
   Get-MoveRequest | ? {$_.Status -eq "Completed"} | Remove-MoveRequest
4. Remove any existing Migration batches: Remove-MigrationBatch “Batch Name” -Force
   

Often a Mailbox move will fail due to item or mailbox level corruption. Allowing for some of the corrupt items to be skipped is often a good way to get a mailbox moved. However, there is the possibility of data loss if you use the below options

1. Open PowerShell (Not via EMS) and connect to Exchange Online.
2. Create a variable to store your on-premises admin credentials. The credentials should be stored in the format of contoso\administrator and not administrator@contoso.com.
   $onpremCred = Get-Credential
3. Then run a cmdlet similar to the following, where ‘User’ is the display name for the account you want to move, ‘Webmail.consoto.com’ is the endpoint that has MRSProxy enabled on-premises (usually this matches the OWA endpoint), and ‘contoso.mail.onmicrosoft.com’ is the routing domain used in Exchange online.
   Example: The following example may result in a minor loss of data since you are allowing some items to be skipped due to corruption:
   New-MoveRequest –Identity ‘User’ -Remote -RemoteHostName 'webmail.contoso.com' -RemoteCredential $onpremCred -TargetDeliveryDomain 'contoso.mail.onmicrosoft.com' –BadItemLimit 40
   

Sorry, we couldn’t resolve your issue with this guide. Please provide feedback on this walkthrough, and then use the resources below to continue troubleshooting. Visit the Office 365 Community for self-help support. Do one of the following:

• Use search to find a solution to your issue.
• Use the Help Center or the Troubleshooting tool that are both available from the top of every community page.
• Sign in with your Office 365 admin credentials, and then post a question to the community.

Visit the Contact Office 365 support page for information about how to submit a support service request.

Often a Mailbox move will fail due to item or mailbox level corruption. Allowing for some of the corrupt items to be skipped is often a good way to get a mailbox moved. However, there is the possibility of data loss if you use the below options

1. Open PowerShell (Not via EMS) and connect to Exchange Online.
2. Create a variable to store your on-premises admin credentials. The credentials should be stored in the format of contoso\administrator and not administrator@contoso.com.
   $onpremCred = Get-Credential
3. Then run a cmdlet similar to the following, where ‘User’ is the display name for the account you want to move, ‘Webmail.consoto.com’ is the endpoint that has MRSProxy enabled on-premises (usually this matches the OWA endpoint), and ‘contoso.mail.onmicrosoft.com’ is the routing domain used in Exchange online.
   Example: The following example may result in a minor loss of data since you are allowing some items to be skipped due to corruption:
   New-MoveRequest –Identity ‘User’ -Remote -RemoteHostName 'webmail.contoso.com' -RemoteCredential $onpremCred -TargetDeliveryDomain 'contoso.mail.onmicrosoft.com' –BadItemLimit 40