Category Archives: Uncategorized

Office 365 – Distribution List Migration Version 2.0 – Part 27

1 Reply

Increasing the performance of multiple distribution list migrations.

In the following blog post I outlined how the multiple distribution list migration process works.

 

Office 365 – Distribution List Migrations Version 2.0 – Part 9 | TIMMCMIC (wordpress.com)

 

In a multiple migration scenario each distribution list in a batch is assigned to a PowerShell job that performs a single distribution list migration. There are certain operations that it is desirable to only perform one time across the migration batch of five groups the main one being Azure Active Directory Connect delta synchronization. The individual distribution list migrations are designed so that all migrations reach a certain point before triggering the operations that we only want performed one time. The first migration thread one is responsible for triggering these operations once threads two through five are ready.

 

There could be any number of issues that cause the distribution list assigned to thread one to fail migration. For example, if a distribution list member is not present in Office 365 the migration of the distribution list will fail as a perquisite exists that all members exist in Office 365 before performing a migration. Originally when thread one failed the performance of the other distribution list migrations were negatively impacted specifically by the lack of triggering of Azure Active Directory Connect delta synchronization. In Azure Active Directory Connect the default sync cycle runs every 30 minutes. If thread one fails and cannot trigger the delta synchronization it could be up to 30 minutes before it would automatically run. This would add 30 minutes to the overall migration time.

 

In version 2.9.5 thread two through five now track the lack of removal of the distribution group from Office 365. If the distribution group is not removed from Office 365 in 5 minutes thread two through five will now attempt a delta synchronization in Azure Active Directory Connect. This cycle repeats itself for every 5 minutes that expire waiting for deletion. It is important to note that there could be service side sync delays which contribute to the amount of time waiting for the distribution list to be deleted from Exchange Online. The assumption is made that if the group is there after 5 minutes it was due to a thread one failure and Azure Active Directory Connect should be triggered.

 

Does this lead to multiple delta sync cycles being run by multiple threads? Generally, not – since each thread has reached the same point in the migration process prior to monitoring Office 365 for the group deletion they will all attempt to trigger Azure Active Directory Connect around the same time. Only one synchronization cycle can proceed at a time therefore one thread will receive a success message that synchronization was triggered, and other threads will receive an error that synchronization is busy. Threads two through 5 do not retry their sync operations but rather simple call the sync process one time. Thread one does have retry logic and always enforces that Azure Active Directory Connect delta sync was triggered one time in the migration process.

By assuming that there is a thread one failure when the distribution list deletion does not occur in a timely fashion and triggering Azure Active Directory Connect we can eliminate undesirable delays in multiple distribution list migrations.

 

Office 365 – Distribution List Migrations Version 2.0 – Part 26

1 Reply

New handling of nested distribution groups in multiple distribution group migrations…

A question that is often asked is how to handle nested distribution list when considering a migration. When performing a single distribution list migration a recipient normalization process scans each member for attributes and object classes. If a group object class is found and is mail enabled the single group migration fails with an exception. The administrator must either first migrate the child distribution group or remove the mail enabled group. Here is a sample exception:

 

ParentGroupSMTPAddress : atestgroup27@domain.com

Alias : aTestGroup270

RecipientOrUser : Recipient

isError : True

isErrorMessage : NestedGroupException – A mail enabled group is a child member of the migrated list. The

child group must be migrated first or removed from group membership.

Name : aTestGroup270

GUID :

OnPremADAttribute : member

GroupType : -2147483640

isAlreadyMigrated : False

ExternalDirectoryObjectID :

DN : CN=aTestGroup270,OU=MigrationTest,OU=DLConversion,DC=home,DC=domain,DC=com

RecipientType :

OnPremADAttributeCommonName : Member

PrimarySMTPAddressOrUPN : aTestGroup270@domain.com

 

For customers that were evaluating options of migration distribution lists in bulk my original advice was to generate the list of primary SMTP addresses for migration. This list would then be run through the multiple migration process. In the end any group that could be migrated was migrated and any group left had an exception. The same set of addresses would be processed multiple times until no more migrations completed. The lists that were left required administrative intervention. The process overall was highly inefficient and sorting through the errors could be complicated. For some customers it was easier to iterate through the list rather than try to outline the dependencies and perform the migrations in order.

For customers that were able to map nested group membership the addresses would be sorted in correct order. The multiple migration process was called on the list and as long as there were no overlap in how the addresses were batched into groups of 5 the migrations would proceed successfully. Nested DLs were migrated first and the parent DLs later in the batch. If there was any overlap, for example a parent and child included in the same batch of simultaneous migrations, the parent would fail with the exception above.

For a review of how the module tracks nested membership and groups that have already been migrated see the following post. Office 365 – Distribution List Migrations Version 2.0 – Part 6 | TIMMCMIC (wordpress.com)

In version 2.9.5 new code is implemented in start-multipleDistributionListMigration that no longer requires the administrator to iterate through the list multiple times or pre-sort the list to account for nesting. The new code implements the following process:

  • Iterate through the list and attempt to migrate each distribution list specified.
  • If the start-DistributionListMigration encounters a nested group exception add the groups normalized information to a CSV file.
  • At the end of the first pass start-MultipleDistributionListMigration tests for the csv file presence.
    • If the CSV file is present:
      • The recipient objects are imported.
      • The child distribution list primary SMTP address is searched in the array of recipients initially migrated.
      • If the child primary SMTP address is found the parent list is added again for migration.
      • If the child primary SMTP address is not found the parent is ignore with a permanent exception. The child was not originally included in the migration set.
      • Scan all groups to ensure there are no cross dependencies. If a cross dependency is found fail the group migration.
    • If the CSV file is not present the migration is complete.
  • If the recipient array is populated with parent group SMTP addresses the multiple list migration is restarted using the concatenated list.
  • The process continues to repeat until there are no more groups to retry.

     

Let examine a practical example of this using the following structure.

 

The array of SMTP addresses that the administrator specified for migration:

GroupA

GroupZ

GroupB

GroupC

GroupB1

GroupC1

GroupC2

GroupC3

GroupC4

GroupX

GroupY

 

The start-MultipleDistributionListMigration code breaks the groups down into the maximum number of threads a single host can migrate. The batch size is 5 groups per batch. The first batch of groups for migration includes:

 

GroupA

GroupZ

GroupB

GroupC

GroupB1

 

At the end of the first batch of migrations GroupB1 migrates successfully as it has no child group dependencies. GroupA, GroupZ, GroupB, and GroupC all fail with nested child groups. These groups are added to the retry list.

 

The next batch of groups includes:

 

GroupC1

GroupC2

GroupC3

GroupC4

GroupX

 

At the end of the second batch of migrations GroupC1, GroupC2, GroupC3, and GroupC4 all migrate successfully as they have no child group dependencies. GroupX fails with a nested child group exception and is added to the retry list.

 

The final batch is a single group – GroupY. GroupY will fail to migrate with a nested child group exception and will be added to the retry list.

 

At the conclusion of all first attempts the retry list contains the following groups:

 

GroupA

GroupZ

GroupB

GroupC

GroupX

GroupY

 

The groups are now scanned to determine if they are eligible for retry.

 

The first can searches the list for circular dependencies. A circular dependency is when two groups are members of each other. In this instance GroupX is a member of GroupY and GroupY is a member of GroupX. This is a cross dependency that cannot be reconciled. These groups hard fail and are added to the failure list for manual administrator intervention.

 

The second scan searches the list of exceptions. Each exception contains the ChildDL primary SMTP address and the ParentDL primary SMTP address. The list of original SMTP addresses is scanned and if the child groups primary SMTP address was present in the list to be migrated then the group is added for retry. Here is an example using our calculated retry list.

 

GroupA@domain.com -> GroupB@domain.com (GroupB@domain.com was originally included add GroupA to the retry list).

GroupA@domain.com -> GroupC@domain.com (GroupC@domain.com was originally included add GroupA to the retry list).

GroupZ@domain.com -> GroupZ1@domain.com (GroupZ1@domain.com was NOT originally included. GroupZ will fail and be added to the failure list for manual administrator intervention).

GroupB@domain.com -> GroupB1@domain.com (GroupB1@domain.com was originally included add GroupB to the retry list).

GroupC@domain.com -> GroupC1@domain.com (GroupC1@domain.com was originally included add GroupC to the retry list).

GroupC@domain.com -> GroupC2@domain.com (GroupC2@domain.com was originally included add GroupC to the retry list).

GroupC@domain.com -> GroupC3@domain.com (GroupC3@domain.com was originally included add GroupC to the retry list).

GroupC@domain.com -> GroupC4@domain.com (GroupC4@domain.com was originally included add GroupC to the retry list).

 

At the end of child validation scanning the list is as follows:

 

GroupA

GroupA

GroupB

GroupC

GroupC

GroupC

GroupC

 

The results are trimmed into a unique list to retry:

 

GroupA

GroupB

GroupC

 

The code then retries the migration using the updated list of addresses to retry. The list contains less than 5 members only a single batch is created. At the conclusion of this second attempt GroupB and GroupC migrate successfully as there are no nested child lists in the group any longer. GroupA fails since at the time of the retry GroupB and GroupC were both members. The process now repeats itself using the new nested exception list.

 

GroupA@domain.com -> GroupB@domain.com (GroupB@domain.com was originally included add GroupA to the retry list).

GroupA@domain.com -> GroupC@domain.com (GroupC@domain.com was originally included add GroupA to the retry list).

 

This results in the following retry list:

 

GroupA

GroupA

 

The results are then trimmed into a unique list to retry.

 

The code then retries the migration using the updated addresses to retry. The list in this case contains only one address so a single batch is created. At the conclusion GroupA now migrates successfully as there are no nested child group dependencies.

 

The migration is concluded at this time.

 

GroupZ will not migrate due to a child group not included in the migration set. Here is an example of an exception for this reason:

 

[1/6/2023 12:20:56 AM] – =====

[1/6/2023 12:20:56 AM] – Alias: a300

[1/6/2023 12:20:56 AM] – Name: a300

[1/6/2023 12:20:56 AM] – PrimarySMTPAddressOrUPN: a300@domain.com

[1/6/2023 12:20:56 AM] – RecipientType: group

[1/6/2023 12:20:56 AM] – GroupType: -2147483640

[1/6/2023 12:20:56 AM] – RecipientOrUser: Recipient

[1/6/2023 12:20:56 AM] – ExternalDirectoryObjectID:

[1/6/2023 12:20:56 AM] – OnPremADAttribute: member

[1/6/2023 12:20:56 AM] – DN: CN=a300,OU=DoNotSync,OU=MigrationTest,OU=DLConversion,DC=home,DC=domain,DC=com

[1/6/2023 12:20:56 AM] – ParentGroupSMTPAddress: aTopZ@domain.com

[1/6/2023 12:20:56 AM] – isAlreadyMigrated: False

[1/6/2023 12:20:56 AM] – isError: True

[1/6/2023 12:20:56 AM] – isErrorMessage: ChildGroupMirationException: The groupt to be migrated has a child group not included in the migration set.

[1/6/2023 12:20:56 AM] – =====

 

GroupX and GroupY will fail to migrate due to a circular dependency. Here is an example of an exception for this reason:

 

[1/6/2023 12:20:56 AM] – =====

[1/6/2023 12:20:56 AM] – Alias: aTopY

[1/6/2023 12:20:56 AM] – Name: aTopY

[1/6/2023 12:20:56 AM] – PrimarySMTPAddressOrUPN: aTopY@domain.com

[1/6/2023 12:20:56 AM] – RecipientType: group

[1/6/2023 12:20:56 AM] – GroupType: -2147483640

[1/6/2023 12:20:56 AM] – RecipientOrUser: Recipient

[1/6/2023 12:20:56 AM] – ExternalDirectoryObjectID:

[1/6/2023 12:20:56 AM] – OnPremADAttribute: member

[1/6/2023 12:20:56 AM] – DN: CN=aTopY,OU=RecursionTest,OU=MigrationTest,OU=DLConversion,DC=home,DC=domain,DC=com

[1/6/2023 12:20:56 AM] – ParentGroupSMTPAddress: aTopX@domain.com

[1/6/2023 12:20:56 AM] – isAlreadyMigrated: False

[1/6/2023 12:20:56 AM] – isError: True

[1/6/2023 12:20:56 AM] – isErrorMessage: CircularReferenceException: This group has a child distribution list that also has this group as a member. This creates a circular dependency which cannot be handeled automatically.

[1/6/2023 12:20:56 AM] – =====

 

What happens if a group fails to migrate for another reason? If a child group fails to migrate due to another error besides a nested group exception the group is not written to the CSV file. This means that when the pass is completed and the CSV file is analyzed – the child DL will no longer be found as included in the migration set. This yields an error that the child group was not present in the migration set. It is quite confusing because it was included in the migration set but unfortunately errored for another reason.

I hope this change allows mass migrations to proceed with greater success and less administrator intervention. Happy migrating!

Office 365 – Distribution List Migrations Version 2.0 – Part 25

1 Reply

Large distribution list migrations and code enhancements to increase performance and reliability.

In part 24 of my blog series, I outlined some of the performance challenges with creating or migrating distribution groups that had large member counts. For testing purposes, I started testing group migrations of 65,000 users. In this post I will review the performance implications associated with migrating or creating groups of this size in Office 365.

In versions of the DLConversionV2 module prior to 2.9.5 the distribution list migration process followed this basic outline:

  • Gather all properties of the distribution list.
  • Normalize all recipients contained in the group or on the properties of the group to their Office 365 identity.
  • Validate that all recipients contained in the group or on the properties of the group were valid in Office 365.
  • Move the group to a non-sync OU, trigger Active Directory replication, and trigger Azure Active Directory connect synchronization.
  • Upon deletion of the group from Office 365 re-create the group.
  • Update all multivalued properties of the group including membership.
  • Update all single valued properties of the group.
  • Conclude with additional mail flow and group updates on premises.

This process works very well for smaller more common groups but introduces several issues with larger groups.

The first challenge was the overall time of the distribution list migration. For a distribution group of 65,000 users the overall migration time was over two days.

Days : 2

Hours : 8

Minutes : 58

Seconds : 9

Milliseconds : 0

Ticks : 2050890000000

TotalDays : 2.37371527777778

TotalHours : 56.9691666666667

TotalMinutes : 3418.15

TotalSeconds : 205089

TotalMilliseconds : 205089000

 

The DLConversionV2 module leverages the Update-DistributionGroupMember commandlet in order to attempt a bulk update. If this command fails for any reason the code automatically reverts to adding each user individually. Knowing that any group of over 2,000 users generally fails the Update-DistributionGroupMember process you can expect large groups to fail the individual member add process. In the case of the group with 65,000 users it took 1.9 days to add all the users to the group.

Days : 1

Hours : 23

Minutes : 40

Seconds : 26

Milliseconds : 0

Ticks : 1716260000000

TotalDays : 1.98641203703704

TotalHours : 47.6738888888889

TotalMinutes : 2860.43333333333

TotalSeconds : 171626

TotalMilliseconds : 171626000

 

I wanted to take a look at the performance of adding an individual member. My observations were that as the migration progressed the time to add a member continued to increase. In early individual user additions the time was between 0 – 2 seconds on each addition and as the migration progressed that time increased to 4 – 5 second per user addition to the group. Here is a sample chart showing the increase over time.

 

The average add was 2.590528 seconds.

The normalization time of converting the users to an Office 365 identity for 65,000 users took slightly under two hours to complete.

Days : 0

Hours : 1

Minutes : 56

Seconds : 58

Milliseconds : 0

Ticks : 70180000000

TotalDays : 0.0812268518518518

TotalHours : 1.94944444444444

TotalMinutes : 116.966666666667

TotalSeconds : 7018

TotalMilliseconds : 7018000

 

 

The health checking and validation of all group dependencies in Office 365 for 65,000 users took slightly under six hours to complete.

Days : 0

Hours : 5

Minutes : 49

Seconds : 45

Milliseconds : 0

Ticks : 209850000000

TotalDays : 0.242881944444444

TotalHours : 5.82916666666667

TotalMinutes : 349.75

TotalSeconds : 20985

TotalMilliseconds : 20985000

 

The concerning time with the large groups was the amount of time to add the members to the group. In the workflow prior to version 2.9.5 it would be almost two days until the group was fully populated and available to users in Office 365. As the group was populating any mail that was received would only be transmitted to those users that were already added resulting in the potential for information to not reach the intended audience.

With these performance characteristics – how has the code improved in version 2.9.5.

In version 2.9.5 changes were made to optimize the total time the group was not wholly available to the users. The overall migration time cannot really be adjusted through code considering the factors outlined previously and in part 24 of my blog series.

In a previous code revision, a change was made to create a new distribution group with a random name as part of the migration process. This group was then renamed and updated as the migration proceeded. This was done to avoid recipient caching that would sometimes cause the creation of the group with the same name to collide with the group that was just removed and fail the migration process. If the random group is the actual target of operations why can we not update the membership of this group prior to removing the original group? There is no reason this cannot be done and this is exactly the change that was made. In version 2.9.5 and newer any changes that are not unique or colliding with the original group are made on this new randomly created group. This allows us to take the operations that take the bulk of time to wholly migrate the group and do them PRIOR to deleting the original group from Exchange Online. The new migration process follows this outline:

  • Gather all properties of the distribution list.
  • Normalize all recipients contained in the group or on the properties of the group to their Office 365 identity.
  • Validate that all recipients contained in the group or on the properties of the group were valid in Office 365.
  • Create the group in Office 365 with a random name.
  • Update all group attributes including membership that do not conflict with the existing group in Exchange Online.
  • Move the group to a non-sync OU, trigger Active Directory replication, and trigger Azure Active Directory connect synchronization.
  • Monitor for deletion of the group from Office 365.
  • Update all remaining multivalued group properties – for example SMTP proxy addresses.
  • Update all single valued properties of the group not previously updated.
  • Conclude with additional mail flow and group updates on premises.

By adjusting the group creation process into two steps instead of one the time that the group is wholly unavailable has been taken from 1.9 days for a group of 65,000 users to 6 minutes!

Days : 0

Hours : 0

Minutes : 6

Seconds : 12

Milliseconds : 0

Ticks : 3720000000

TotalDays : 0.00430555555555556

TotalHours : 0.103333333333333

TotalMinutes : 6.2

TotalSeconds : 372

TotalMilliseconds : 372000

 

When adding users individually it was not uncommon in testing to encounter either a timeout error adding the individual user, or a server return 503 error. A new code change was introduced to capture the error on the first addition attempt and sleep for 15 seconds. At the end of the 15 second sleep a second attempt is made to add the user to the group. If the second attempt fails the user is added to the post creation error tracking and feedback is provided to the administrator that the user must manually be rectified. With a delay implemented and retry logic available this increases the likely hood of success when adding the user. Here is a sample of this new code flow as it is logged.

[1/8/2023 10:37:46 PM] – Attempting to add recipient: 4406ee0c-a2a6-47ab-b6c1-6b08d271da26

[1/8/2023 10:37:47 PM] – Error on individual recipient add.

[1/8/2023 10:37:47 PM] – It is possible that the operation times out or server returns busy – sleep 15 and retry

[1/8/2023 10:37:47 PM] – ********************************************************************************

@{ParameterName=sleepString; Bound=True; ParameterValue=Sleeping due to error on individual add to retry.} @{ParameterName=sleepSeconds; Bound=True; ParameterValue=15} @{ParameterName=sleepParentID; Bound=False; ParameterValue=0} @{ParameterName=sleepID; Bound=False; ParameterValue=0} @{ParameterName=Verbose; Bound=False; ParameterValue=} @{ParameterName=Debug; Bound=False; ParameterValue=} @{ParameterName=ErrorAction; Bound=False; ParameterValue=} @{ParameterName=WarningAction; Bound=False; ParameterValue=} @{ParameterName=InformationAction; Bound=False; ParameterValue=} @{ParameterName=ErrorVariable; Bound=False; ParameterValue=} @{ParameterName=WarningVariable; Bound=False; ParameterValue=} @{ParameterName=InformationVariable; Bound=False; ParameterValue=} @{ParameterName=OutVariable; Bound=False; ParameterValue=} @{ParameterName=OutBuffer; Bound=False; ParameterValue=} @{ParameterName=PipelineVariable; Bound=False; ParameterValue=}

[1/8/2023 10:37:47 PM] – ********************************************************************************

[1/8/2023 10:37:47 PM] – ********************************************************************************

[1/8/2023 10:37:47 PM] – BEGIN start-sleepProgess

[1/8/2023 10:37:47 PM] – ********************************************************************************

[1/8/2023 10:38:02 PM] – END start-sleepProgess

[1/8/2023 10:38:02 PM] – ********************************************************************************

[1/8/2023 10:38:02 PM] – Error procesing recipient: 4406ee0c-a2a6-47ab-b6c1-6b08d271da26

Ex94914C|Microsoft.Exchange.Configuration.Tasks.ManagementObjectNotFoundException|Couldn’t find object “4406ee0c-a2a6-47ab-b6c1-6b08d271da26”. Please make sure that it was spelled correctly or specify a different object.

@{Name=aTestGroup383; Alias=aTestGroup383; PrimarySMTPAddressorUPN=aTestGroup383@e-mcmichael.com; ExternalDirectoryObjectID=49781059-af43-45e1-877c-6bee00a41af4; ErrorMessageDetail=Ex94914C|Microsoft.Exchange.Configuration.Tasks.ManagementObjectNotFoundException|Couldn’t find object “4406ee0c-a2a6-47ab-b6c1-6b08d271da26”. Please make sure that it was spelled correctly or specify a different object.; Attribute=Cloud Distribution Group Member; ErrorMessage=Member 4406ee0c-a2a6-47ab-b6c1-6b08d271da26 unable to add to cloud distribution group. Manual addition required.}

In this example the recipient was deleted prior to the membership addition and the failure is accurate both on the first and second add. The operation is aborted and the administrator is notified.

Overall, the changes implemented in 2.9.5 increase not only the performance and reliability of large group migrations but also smaller group migrations.

 

Office 365 – Distribution List Migrations Version 2.0 – Part 24

1 Reply

Issues encountered with large distribution list migrations…

I recently had a conversation with a customer that was looking at options for migrating distribution lists to Office 365. In many cases they had several distribution lists that had well over 20,000 members and some with over 100,000 members. These lists were quite large by comparison to many of the customers I’ve worked with as well as my test cases. With this request I hit up the lab and started doing some testing.

A common method for updating distribution group membership is using the Update-DistributionGroupMember commandlet. This commandlet takes an array of user objects and overwrites the existing group membership of the specified group. There is convenience in using this command in that administrators do not have to iterate though a group of members in order to reset a groups membership. In my testing I observed some interesting results as it pertained to large distribution groups.

When using Update-DistributionGroupMember to reset group membership there was a high success rate using recipient array sizes of 0 – 2000. When you started hitting recipient counts greater than 2000 the success of the operation was mixed at best. For example, in testing groups 0 – 2000 the command would complete with a success return. When using recipient array sizes of 2000 – 4000 the result would be a timeout error, yet the membership addition was successful. Any attempts to update membership with arrays greater than 4000 resulted in the command timing out and no membership updates occurred. In essence any attempt to modify the membership using this commandlet above 2000 recipients required validation of success and / or iterating through the recipient array and adding the user individually. Unfortunately, the Update-DistributionGroupMember command is not a add command, therefore recipient arrays cannot be batched and the command called more than one.

The success or failure of Update-DistributionGroupMember is directly tied to the time of processing within the service. When the service is attempting to process these membership operations there is a 15 minute timeout. Anytime it takes greater than 15 minutes to process the membership array the service will abort and timeout the command. It is not uncommon with larger group modifications you hit this limit. In the scenario where a portion of the users were processed we hit a different timeout. In this instance the service was in the midst of processing the operation and the client itself timed out the session. The client times out a request after 16 minutes. In this case the service did not respond back with an affirmative processing of all the users and the client timed out the operation. As indicated above the sweet spot seemed to be around the 4000 user mark for overall success and 2000 for absolute success.

Contributing to the 15 minute timeout are Active Directory lookups in the Exchange Online directories for each recipient passed either individually or in the Update-DistributionGroupMember commandlet. When calling Update-DistributionGroupMember each member passed has an individual lookup on them validating their existence and other information necessary to complete the membership update.

The other factor that contributes to the migration performance is the dual write aspects within Exchange Online. In the new architecture when a recipient operation occurs though the Exchange Online Powershell or Exchange Online control panel the operation is simultaneously written to both the Exchange Online directory and Azure Active Directory at the same time. This process is known as dual write. Overall, this is a good thing. In the past when there was both a forward sync and backsync process sometimes changes were not efficiently transmitted to both directories in a timely fashion. Dual write largely eliminates this issue but also adds additional overhead of performing simultaneous operations in two locations. The benefits far outweigh the additional time in this author’s opinion but it does add overhead to not only the Update-DistributionGroupMember process but also the individual recipient addition process.

In part 25 of my blog series I’ll look at the migration performance of a group of 65,000 users and the code changes taken to improve this experience.

Office 365 – Distribution List Migration – Version 2.0 – Table of Contents

5 Replies

Interested in content regarding Office 365 and Distribution List Migrations. The following table of contents links to relevant articles regarding the Distribution List Migration Module Version 2.

Office 365 – Distribution List Migrations Version 2.0 – Part 2

2 Replies

Preparing to use the distribution list migration v2 module.

The distribution list migration module version 2 is now a published module in the PowerShell gallery. The module has several external dependencies required for it to complete a migration. These dependencies include the Exchange Online Management Module V3, the Azure Active Directory PowerShell Module, the TelemetryHelper PowerShell Module, and the Remote Server Administration Tools for Active Directory.

 

Administrator rights to the workstation or server where the migrations will be performed are also required to establish the initial pre-requisites.

 

The DLConversionV2 PowerShell Module now includes the necessary modules in the manifest file. When the DLConversionV2 module is installed all of the dependent modules available from the PowerShell Gallery are also installed. The script ensures the minimum tested versions of the modules are present.

 

RequiredModules = @(

    @{ModuleName = ‘ExchangeOnlineManagement’; ModuleVersion = ‘3.0.0’ },

    @{ModuleName = ‘AzureAD’; ModuleVersion = ‘2.0.2.140’},

    @{ModuleName = ‘TelemetryHelper’; ModuleVersion = ‘1.5.1’}

    ‘ActiveDirectory’

)

 

 

Administrators should use the UPDATE-MODULE command to ensure that each of the dependent modules are up to date. The DLConversionV2 module is tested and released using the latest versions of dependent modules.

 

To begin preparing to utilize the module open PowerShell as an administrator. The execution policy must be relaxed in order to allow the utilization of unsigned modules.

 

SET-EXECUTIONPOLICY UNRESTRICTED

 

The installation of the Exchange Online Management Module V2 is performed through PowerShell.

About the Exchange Online PowerShell V2 module | Microsoft Docs

 

INSTALL-MODULE EXCHANGEONLINEMANAGEMENT

OR

UPDATE-MODULE EXCHANGEONLINEMANAGEMENT

 

As of the time of publishing, the script was tested with Exchange Online Management Module v 2.4.

 

In order to interact with Active Directory, the Remote Server Administration Tools for Active Directory are required. If your client is a Windows Server the tools can be installed through PowerShell.

 

INSTALL-WINDOWSFEATURE RSAT-ADDS

 

If your client is a Windows 10 workstation, see the following to determine the correct installation instructions for your version of Windows 10.

Remote Server Administration Tools | Microsoft Docs

 

The final step is to install the DLConversionV2 PowerShell module. As with the Exchange module this module can either be installed of updated.

 

INSTALL-MODULE DLCONVERSIONV2

OR

UPDATE-MODULE DLCONVERSIONV2

 

With the module and pre-requisites installed, you are now ready to explore the commands for migrating distribution groups.

 

Command included with the module can be viewed with:

 

GET-COMMAND -MODULE DLCONVERSIONV2

 

Help for commands within the module can be viewed with:

 

GET-HELP START-DISTRIBUTIONLISTMIGRATION (OR OTHER COMMAND)

 


 

Office 365 – Distribution List Migration Version 2.0 – Part 23

2 Replies

Collecting telemetry for performance improvements and usage statistics.

Version 2.8.0 is now available for installation and provides telemetry submission on the performance of distribution list migrations. Two new switches have been added to the powershell module:

-allowTelemetryCollection

-allowDetailedTelemetryCollection

 

What data is included with -allowTelemetryCollection?

  • CustomDimensions
    • AzureAD Powershell Module Version
    • DLConversionV2 command executed
    • DLConversionV2 Powershell Module Version
    • ExchangeOnline Powershell Module Version
    • Migration Start Time UTC
    • Migration End Time UTC
    • True / False errors encountered during the migration
    • OS Version executing the script

  • CustomMetrics (Time recorded in seconds)
    • Migration elapsed time
    • Time pending DL removal in Office 365
    • Time to collect dependencies in Office 365
    • Time to collect dependencies on premises
    • Time to create the Office 365 DL and update attributes
    • Time to normalize all on premises dependencies
    • Time to validate and perform safety check on DL members
    • Time to replace on premises dependencies post migration
    • Time to replace Office 365 dependencies post migration

 

 

What data is included with -allowDetailedTelemetryCollection?

  • CustomDimensions
    • Same items as standard telemetry.

 

  • CustomMetrics
    • All items included in standard telemetry.
    • All recipient counts including members, number of users on attributes, and all counts of dependencies on premises and in Office 365.

 

 

Does the module collect any personally identifiable information?

The module does not collect any personally identifiable information, this includes hostnames and IP addresses. Azure App Analytics does collect the city, state, and country where the submission is made but no specific geo-location data is included beyond this information.

 

 

What will you do with this information?

The performance information will allow me to monitor the efficiency of the script. I hope to utilize the data to continue to create real world scenarios for my testing purposes. In general the count of usage will help drive further development efforts of the module.

 

Can I opt out of telemetry collection?

Absolutely! In default configuration telemetry collection is automatically enabled for each run. Should you desire to opt out of any form of telemetry collection please specify -allowTelemetryCollection:$FALSE in the command execution. If you desire to opt out of detailed telemetry collection but still provide basic information use -allowDetailedTelemetryCollection:$FALSE

Office 365 – Distribution List Migration Version 2.0 – Part 22

1 Reply

The end of basic authentication in Exchange Online…

Fall is now upon us and as we enter the month of October things are changing in Exchange Online. I hope that most of my administrative peers are aware that basic authentication in Exchange Online will be disabled October 1st, 2022.

The preparations for this have been years in the making. I’m personally excited about the enhanced security this will bring to our customers. One of the preparation items was to prepare and release a version of the Exchange Online Management Module that no longer requires the use of basic authentication. Many administrators did not realize but even prior versions of the module that “supported modern authentication” were still leveraging basic authentication under the hood as part of the connection to Exchange Online.

Exchange Online Management Module version 3.0.0 is now published to the PowerShell gallery. With the introduction of this module administrators now have a fully modern authentication-based method of performing Exchange Online management through PowerShell. In addition, this module moves away from the legacy PowerShell infrastructure to new REST based commands. These new commands have the same look and feel without the issues that the legacy remote PowerShell infrastructure had.

Distribution List Migration module version 2.7 is now hard coded to require a minimum version of 3.0.0 to perform distribution list migrations. Administrators may upgrade to this version using the update-module ExchangeOnlineManagement command or install the module for the first-time using install-module ExchangeOnlineManagement.

Happy migrating!

Office 365 – Distribution List Migration Version 2.0 – Part 21

1 Reply

Preparing for the deprecation and disablement of Basic Authentication.

October 2022 is fast approaching and the topic of disabling basic authentication in Exchange Online is a hot topic. If you were not aware, starting in October, Microsoft will begin proactively disabling Basic Authentication across the Exchange Online platforms. This is a good thing and is long overdue.

 

In addition to Exchange Online disabling basic authentication many customers are also starting to disable basic authentication in the WinRM client within windows. This effectively blocks any use of basic authentication for remote PowerShell connections even if basic authentication is still enabled in Office 365. (Note: Group policies for disabling WinRM client basic authentication does not disable basic authentication in Exchange Online).

 

When customers disable basic auth in the WinRM client connections to Exchange Online and the Exchange On-Premises PowerShell using basic authentication will subsequently fail. For example, when using the Exchange Online Management PowerShell version 2.0.5 you will observe the following error:

 

New-ExoPSSession : Create Powershell Session is failed using OAuth

At C:\Program

Files\WindowsPowerShell\Modules\ExchangeOnlineManagement\2.0.5\netFramework\ExchangeOnlineManagement.psm1:475 char:30

+ … PSSession = New-ExoPSSession -ExchangeEnvironmentName $ExchangeEnviro …

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : NotSpecified: (:) [New-ExoPSSession], Exception

+ FullyQualifiedErrorId : System.Exception,Microsoft.Exchange.Management.ExoPowershellSnapin.NewExoPSSession

 

You might be asking yourself why does this happen if version 2.0.5 supports modern authentication? Even though this version of the PowerShell management module supports modern authentication, it does so by obtaining a token on behalf of the administrator and then encapsulating the token into the basic auth session that pulls the commandlets from remote PowerShell. It acted as a middle ground to help bring better authentication to PowerShell while still using the legacy remote PowerShell interfaces.

 

If an administrator attempts to open a basic authentication through new-pssession to Exchange On-Premises, the following error is noted in absence of basic authentication in the WinRM client.

 

New-PSSession : [server.domain.com] Connecting to remote server server.domain.com failed with the

following error message : The WinRM client cannot process the request. Basic authentication is currently disabled in

the client configuration. Change the client configuration and try the request again. For more information, see the

about_Remote_Troubleshooting Help topic.

At line:1 char:12

+ $session = New-PSSession -ConfigurationName microsoft.exchange -Conne …

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin

gTransportException

+ FullyQualifiedErrorId : -2144108321,PSSessionOpenFailed

 

As an administrator you can validate the WinRM authentication configuration using the following command:

 

C:\>winrm get winrm/config/client/auth

Auth


Basic = false
[Source=”GPO”]

Digest = true

Kerberos = true

Negotiate = true

Certificate = true

CredSSP = false

 

When originally released the DLConversionV2 module leveraged basic authentication for both Exchange Online and Exchange On-Premises commands. With the disablement of basic authentication pending the script has been updated and validated to move beyond basic authentication.

 

For Exchange Online to move beyond basic authentication administrators must install the Exchange Online Management Module preview version 2.0.6. This module supports modern authentication, removes the requirement for basic authentication, and takes full advantage of the new REST based commandlets for better performance. To begin the update process:

 

First ensure that powershell get is updated so that you may take advantage of the allow pre-release function:

 

Update-Module PowerShellGet -force <or> Install-Module PowerShellGet -force if the module was not installed by repository

 

Exit and reopen Windows Powershell to ensure the new modules are loaded.

 

Second install the Exchange Online Management preview module:

 

Install-Module ExchangeOnlineManagement -allowPreRelease <or> Update-Module ExchangeOnlineManagement -allowPreRelease

 

You may validate the success of the installation by running the following command and validating the version 2.0.6.

 

PS C:\> Get-Command -Module ExchangeOnlineManagement

 

CommandType Name Version Source

———– —- ——- ——

Function Connect-ExchangeOnline 2.0.6
ExchangeOnlineManagement

 

When using the DLConversionV2 migration module with any of the pre-collection commands for on-premises, the enable-hybridMailFlowPostMigration command, or performing a distribution list migration with enablehybridMailFlow=$TRUE the administrator must specify a different authentication method besides authentication. The -exchangeAuthentication method allows the administrator to specify the use of Kerberos authentication. In addition to utilizing an Exchange authentication method the Exchange Server specified is typically a named server and not a name associated with a load balanced group of Exchange servers. Here is an example of a sample command:

 

Start-DistributionListMigration -groupSMTPAddress test@contoso.com -globalCatalog dc.contoso.com -activeDirectoryCredential $cred -exchangeServer mail-server-a.contoso.com -exchangeCredential $cred -exchangeAuthenticationMethod:Kerberos -exchangeOnlineCredential $cred -azureADCredential $cred -enableHybridMailflow:$TRUE -dnNoSyncOU: “OU”

 

In addition to the DLConversionV2 modules 2.6.0 or newer supporting the preview PowerShell module and Kerberos it also supports certificate authentication for both Exchange Online and Azure Active Directory. This may also serve as an alternative to specifying credentials as a part of the migration process.

 

These changes and tested scenarios should allow you to transition away from basic authentication while allowing your migrations to be successful.

Office 365 – Distribution List Migration Version 2.0 – Part 20

3 Replies

Adding a new method of verifying the distribution list is directory synchronized.

When a distribution list is migrated to Office 365 there are several health check pre-requisites that are performed to ensure the migration will be successful. One of the checks that is performed is to ensure that the list being migrated is directory synchronized to Office 365.

 

The check is performed by capturing the distribution list configuration from Exchange Online and reviewing the flag isDirSynced. If the flag has a value of TRUE, this signifies that the group is directory synchronized and is eligible for migration. If the value is FALSE this would typically mean the group is not directory synchronized.

 

PS C:\> Get-DistributionGroup atestgroup200 | fl isDirSynced

IsDirSynced : True

 

In Exchange Online there is a forward synchronization process that is responsible for replicating attributes and changes from Azure Active Directory to the corresponding objects in Exchange Online. It was recently discovered that in some circumstances during the forward synchronization process a distribution list that is directory synchronized would be shown as not directory synchronized.

 

PS C:\> Get-DistributionGroup TestDirSync2 | fl isDirSynced

IsDirSynced : False

 

When a distribution list migration is performed under these circumstances the migration process fails. This is by design; the migration process should fail anytime a group migration is attempted and the group is not directory synchronized. The obvious problem is that the group IS directory synchronized.

 

This situation highlights that there could be additional factors that cause migrations to fail when they should not. The real source of authority for group information is Azure Active Directory. In version 2.6.0 of the DLConversionV2 migration module an additional requirement is now present to connect to Azure Active Directory and validate group information. In the function where directory synchronization is verified if the flag in Exchange Online is set to false for any reason the information from Azure Active Directory is validated. If the information demonstrates the group is directory synchronized the migration is allowed to proceed. If the information verifies that the group is not directory synchronized the migration fails. Here is a sample of the same group above from Azure Active Directory.

 

PS C:\> Get-AzureADGroup -SearchString TestDirSync2 | fl DirSyncEnabled

DirSyncEnabled : True

 

With version 2.6.0 there are additional installation requirements. The powershell module for Azure Active Directory must now be used.

 

Install-Module AzureAD

 

When invoking a distribution list migration, you must now either specify credentials to connect to Azure Active Directory or you must specify the certificate authentication information necessary to complete the connection.

 

To specify credentials use -AzureADCredential $cred -azureEnvironemnt <Name> (Optional: Used when connecting to a non-commercial azure environment)

 

To specify certificate authentication use -azureTenantID “TenantID” -azureApplicationID “AppID” -azureCertifcateThumbprint “Thumbprint” -azureEnvironment <name> (Optional: Used when connecting to a non-commercial azure environment)

 

In most cases that same credential utilized for Exchange Online should have sufficient permissions for this operation. If there is a split delegation model the account specified for Azure AD would have to have permissions to read group information and execute the get-AzureADGroup commandlet.