Issues encountered with large distribution list migrations…

I recently had a conversation with a customer that was looking at options for migrating distribution lists to Office 365. In many cases they had several distribution lists that had well over 20,000 members and some with over 100,000 members. These lists were quite large by comparison to many of the customers I’ve worked with as well as my test cases. With this request I hit up the lab and started doing some testing.

A common method for updating distribution group membership is using the Update-DistributionGroupMember commandlet. This commandlet takes an array of user objects and overwrites the existing group membership of the specified group. There is convenience in using this command in that administrators do not have to iterate though a group of members in order to reset a groups membership. In my testing I observed some interesting results as it pertained to large distribution groups.

When using Update-DistributionGroupMember to reset group membership there was a high success rate using recipient array sizes of 0 – 2000. When you started hitting recipient counts greater than 2000 the success of the operation was mixed at best. For example, in testing groups 0 – 2000 the command would complete with a success return. When using recipient array sizes of 2000 – 4000 the result would be a timeout error, yet the membership addition was successful. Any attempts to update membership with arrays greater than 4000 resulted in the command timing out and no membership updates occurred. In essence any attempt to modify the membership using this commandlet above 2000 recipients required validation of success and / or iterating through the recipient array and adding the user individually. Unfortunately, the Update-DistributionGroupMember command is not a add command, therefore recipient arrays cannot be batched and the command called more than one.

The success or failure of Update-DistributionGroupMember is directly tied to the time of processing within the service. When the service is attempting to process these membership operations there is a 15 minute timeout. Anytime it takes greater than 15 minutes to process the membership array the service will abort and timeout the command. It is not uncommon with larger group modifications you hit this limit. In the scenario where a portion of the users were processed we hit a different timeout. In this instance the service was in the midst of processing the operation and the client itself timed out the session. The client times out a request after 16 minutes. In this case the service did not respond back with an affirmative processing of all the users and the client timed out the operation. As indicated above the sweet spot seemed to be around the 4000 user mark for overall success and 2000 for absolute success.

Contributing to the 15 minute timeout are Active Directory lookups in the Exchange Online directories for each recipient passed either individually or in the Update-DistributionGroupMember commandlet. When calling Update-DistributionGroupMember each member passed has an individual lookup on them validating their existence and other information necessary to complete the membership update.

The other factor that contributes to the migration performance is the dual write aspects within Exchange Online. In the new architecture when a recipient operation occurs though the Exchange Online Powershell or Exchange Online control panel the operation is simultaneously written to both the Exchange Online directory and Azure Active Directory at the same time. This process is known as dual write. Overall, this is a good thing. In the past when there was both a forward sync and backsync process sometimes changes were not efficiently transmitted to both directories in a timely fashion. Dual write largely eliminates this issue but also adds additional overhead of performing simultaneous operations in two locations. The benefits far outweigh the additional time in this author’s opinion but it does add overhead to not only the Update-DistributionGroupMember process but also the individual recipient addition process.

In part 25 of my blog series I’ll look at the migration performance of a group of 65,000 users and the code changes taken to improve this experience.