Exchange Server and Office 365 offer many different options for forwarding messages to different recipients. Some of these options exist for users and others are for administrators. Administrators can also control how forwarding is handled within the organization.

 

The forwarding options available to clients and administrators are described below. Each of these methods has both pros and cons when implemented:

 

• Users can create a forwarding rule within Outlook or Outlook Web App
• An administrator can create a client rule for forwarding
• An administrator can set the ForwardingSMTPAddress parameter on a mailbox
• An administrator can set the ForwardingAddress parameter on a mailbox

 

In this post, I’m going to review each of these methods, the pros and cons, and the administrator controls in place for these options.

 

Forwarding rule within Outlook and Outlook Web App:

Users can create Inbox rules in both Outlook and Outlook Web App (OWA). One of the options is a rule to forward messages to a different recipient, which can be an internal or external recipient. In the example below, a rule has been created that forwards all messages received in the mailbox to an external recipient.

 

 

Client-side rules allow a great deal of flexibility when establishing forwarding. For example, say you want to forward emails to another internal or external recipient based on who sent the messages. The rules wizard allows this level of granularity. From an administrative standpoint client-side rules can also be helpful since they can be managed by the user. This obviates the need for administrators to be involved in the day-to-day management of these rules.

 

Client-side rules though are not immediately obvious to administrators. This may cause issues in environments that want to control where their data is stored and how it is transmitted. No specific permissions are required for users to create rules and there exists no method to limit this type of rule. Messages received by the destination mailbox look like forwarded messages; they have the same presentation as if the end-user selected the forward option on the message and specified the recipient.

 

 

 

Message tracking also confirms that the sender is the mailbox with the forwarding rule enabled and the recipient is the address specified on the forwarding rule.

 

 

 

This may cause issues in message handling, as subsequent replies would go to the mailbox the message was forwarded from rather than the original sender of the message.

 

When a message is forwarded by a client-side rule an additional header is added to the message: X-MS-Exchange-Organization-AutoForwarded. When the value of this header is set to TRUE this signifies the message was created by auto-forwarding.

 

Forwarding rule created by the administrator:

In Exchange 2010 and Exchange 2013, administrators have the New-InboxRule cmdlet, which allows them to create client-side rules within a mailbox. To create a forwarding rule that matches the previous rules wizard example you would run a command similar to the following:

 

PS C:> New-InboxRule -Name ForwardSharon -Mailbox 2148 -ForwardTo:SHARON@domain.com

Name                          Enabled                       Priority                      RuleIdentity
—-                          ——-                       ——–                      ————
ForwardSharon                 True                          1                             11219471045587107842

 

When using the Outlook client and viewing the rules in this mailbox, the administrator-created rule is present.

 

 

This rule has the same considerations and outcomes as a rule created by a user in Outlook; the only difference is that an administrator added it to the mailbox.

 

Forwarding using the forwardingSMTPAddress parameter of a mailbox:

Each mailbox has a parameter named forwardingSMTPAddress. This parameter is set by the administrator using Set-Mailbox. Any SMTP address can be specified in this parameter.

 

PS C:> Set-Mailbox 2148 -ForwardingSmtpAddress sharon@domain.com

PS C:> Get-Mailbox 2148 | fl name,forwardingSMTPAddress,delivertomailboxandforward

Name                       : Tim McMichael
ForwardingSmtpAddress      : smtp:sharon@domain.com
DeliverToMailboxAndForward : False

When this type of forwarding is used, the user has no control and the forwarding process must be administratively managed. In addition, there is no granularity with this implementation; all messages for the mailbox are forwarded to the specified SMTP address. Because the field allows any SMTP address to be specified there is no need to create a mail-enabled object within your directory to establish this type of forwarding.

 

Messaging handling is also different with this type of forwarding. When the message arrives in the destination mailbox it appears as if it was sent directly to this mailbox.  The TO field of the message reflects the original recipient, and the FROM field reflects the original sender. The message does not appear as forwarded. 

 

 

Message tracking shows that the sender of the message is the original sender and the recipient is the forwarding address.

 

 

This allows the recipient to reply to the message and have the message sent directly to the original sender. It is important to understand this behavior in scenarios where compliance or journaling is used, as subsequent replies will bypass the original organization thereby bypassing journaling and compliance processes.

 

The message header also has appended the X-MS-Exchange-Organization-AutoForwarded header. When the value of this header is set to TRUE, in indicates that the message was created by auto-forwarding.

 

Forwarding using the forwardingAddress parameter of a mailbox:

The forwardingAddress parameter of a mailbox is also managed by an administrator using Set-Mailbox. Unlike forwardingSMTPAddress, the forwarding address parameter must be an object that is defined within the directory. Attempting to set this value to an object not within the directory returns an error.

 

PS C:> Set-Mailbox 2148 -ForwardingAddress sharon@domain.com
Couldn’t find object “sharon@domain.com”. Please make sure that it was spelled correctly or specify a different
object.
    + CategoryInfo          : NotSpecified: (:) [Set-Mailbox], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=BN1PR06MB101,RequestId=3374349c-1924-43b2-bb21-d2243774a1d4,TimeStamp=7/20/2014
   5:37:29 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] F2A1EE0E,Microsoft.Exchange.Management.Reci
  pientTasks.SetMailbox
    + PSComputerName        : pod51043psh.outlook.com

 

The forwarding address parameter can be another mailbox-enabled object, a mail-enabled user, or a mail contact.  In this example, I have created a mail contact for my external recipient. 

 

PS C:> New-MailContact -Name Sharon -ExternalEmailAddress sharon@domain.com

Name                      Alias                                          RecipientType
—-                      —–                                          ————-
Sharon                    Sharon                                         MailContact

 

I then enabled the forwardingAddress parameter on the mail contact.

 

PS C:> Set-Mailbox 2148 -ForwardingAddress Sharon

 

The setting was validated with Get-Mailbox.

 

PS C:> Get-Mailbox 2148 | fl name,forwardingAddress,delivertomailboxandforward

Name                       : Tim McMichael
ForwardingAddress          : Sharon
DeliverToMailboxAndForward : False

 

Similar to the fowardingSMTPAdderss parameter the forwardingAddress parameter must be managed by the administrator. The end-user has no control over this specific parameter and the end-user will not know that forwarding is enabled. Unlike client rules, the use of the forwardingAddress parameter applies to all messages received to the destination mailbox; there is no way to allow this function to work on specific messages.

 

Messaging handling is also similar to using the fowardingSMTPAddress parameter. The message arrives in the destination mailbox as if it was addressed directly to that mailbox. The TO address shows the original recipient and the FROM address shows the original sender. 

 

 

This allows the forwarded recipient to reply to the message and have it returned directly to the original sender. Messaging tracking also shows that the sender is the original sender and the recipient is the address configured for forwarding.

 

 

It is important to understand this behavior in scenarios where compliance or journaling is used as subsequent replies will bypass the original organization thereby bypassing journaling and compliance processes.

 

In this scenario the X-MS-Exchange-Organization-AutoForwarded header is not stamped on the message.

 

Autoforwarding and delivertoandforward:

Each mailbox also has a parameter called deliverToAndForward. This allows the administrator to specify if the message should also be delivered to the mailbox where forwarding is enabled. This setting applies ONLY to forwarding that is configured by using the forwardingAddress or forwardingSMTPAddress parameters. Client-side rules are unaware of this setting.

 

Autoforwarding and remoteDomains:

Within Exchange and Office 365, administrators can create remote domains. By default, every tenant leverages the ‘*’ domain. When a specific remote domain does not exist, the ‘*’ remote domain setting are applied to the message.

 

A property of a remote domain is autoForwardEnabled property. This allows administrators to define if auto-forwarding is allowed on messages destined to the domain specified.

 

PS C:> Get-RemoteDomain  | fl name,domainname,autoForwardEnabled

Name               : Default
DomainName         : *
AutoForwardEnabled : True

 

By default auto-forwarding is allowed to all domains. Administrators can use the remote domain settings to control how forwarding outside their organization is handled. Please note: this does not change how forwarding is handled within the organization.

 

In this example, I am creating a specific remote domain rule and have disabled auto-forwarding. 

 

PS C:> New-RemoteDomain -Name ExternalDomain -DomainName domain.com

Name                           DomainName                                   AllowedOOFType
—-                           ———-                                   ————–
ExternalDomain                 domain.com                                   External

 

PS C:> Set-RemoteDomain -Identity ExternalDomain -AutoForwardEnabled:$FALSE

 

The settings can be verified with get-remoteDomain:

 

PS C:> Get-RemoteDomain ExternalDomain | fl domainname,autoforwardenabled

DomainName         : domain.com
AutoForwardEnabled : False

 

To test this feature I set the forwardingAddress on the mailbox to a recipient in the domain.com domain. When a message is addressed to the mailbox where forwarding is enabled, it never makes it to the destination. The same behavior would occur if a client-side forwarding rule is used.

 

When a client-side rule or the forwardingSMTPAddress is used, the mail flow process stamps the X-MS-Exchange-Organization-AutoForwarded to TRUE. This header is not viewable on a message as the transport header firewall removes it. When the message is processed by the Transport service, if the remote domain specifically blocks auto-forwarding and the auto-forward header is present and set to TRUE, the message is turfed, and no NDR is generated. If no form of journaling is present and the deliverToAndForward setting of the mailbox is set to FALSE, the message is effectively lost.

 

When looking at message tracking we can see that the message failed to reach its destination. The message delivery status is NONE and no send events are noted.

 

 

 

Using the same remote domain, the mailbox is configured with a forwardingAddress instead. The mail contact used as the forwardingAddress object has an external email address in domain.com. When sending a message to the mailbox where forwarding is enabled, the mail is successfully delivered. This is also confirmed by message tracking.

 

 

In this example, the transport process does not use the X-MS-Exchange-Organization-AutoForwarded but instead adds the forwarded address as a recipient during a different stage of message processing. As the message traverses the transport stack it is not detected as a forwarded message and is therefore allowed to arrive at desired recipient.

 

Utilizing the forwardingAddress scenario is an excellent way for an administrator to bypass forwarding settings on remote domains while controlling when message forwarding is allowed.

 

We routinely receive requests on how to enable granular message forwarding or only allow specific users to use auto-forwarding capabilities. Unfortunately, with the forwarding settings being tied to remote domains it is very difficult to enable forwarding for only a subset of users or where granular forwarding on messages attributes is desired. To allow clients to create auto-forwarding rules would mean the remote domain would have to allow auto-forwarding for everyone. Administrators can attempt to discover and remove auto-forwarding rules using Get-InboxRule. 

 

In my experience, in most scenarios forwarding is disallowed on remote domains and enabling of forwarding is restricted to administrator configuration using the forwardingAddress parameter.

The Start-DatabaseAvailabilityGroup cmdlet is used to restore failed nodes in a Database Availability Group (DAG) by joining the evicted nodes back to the existing cluster. When a node joins a cluster it is assigned a unique decimal value ID.  For example, the first node is assigned Node ID 1, the second node Node ID 2, and so on up to 16 supported nodes in a DAG.  You can view the node ID assigned to a particular node via the registry.  Navigate to HKLM Cluster Nodes:

 

 

The subkeys are the decimal values assigned to each node.  Selecting a subkey will expose the NetBIOS node name assigned to the NodeID.  In this example Node ID 1 is assigned to server MBX-1.

 

 

The servers list of the DAG is a multi-valued attribute that represents the members of the DAG. 

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers

Name    : DAG
Servers : {MBX-4, MBX-3, MBX-2, MBX-1}

 

When Stop-DatabaseAvailabilityGroup is run, the task parses the servers list and determines if an action should be taken on that server. For example, if the next server on the list falls within the AD site being stopped, the cmdlet will take action on that server. 

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers,startedmailboxservers,stoppedmailboxservers

Name                  : DAG
Servers               : {MBX-4, MBX-3, MBX-2, MBX-1}
StartedMailboxServers : {MBX-3.domain.com, MBX-4.domain.com}
StoppedMailboxServers : {MBX-2.domain.com, MBX-1.domain.com}

 

After running Restore-DatabaseAvailabilityGroup, the nodes on the stopped servers list are evicted from the cluster.  This results in the Node IDs within the cluster registry associated with those nodes being freed. 

 

 

At this stage, the secondary site is now functional, independent of the original.  Hopefully there will be a time where the primary site is accessible and its nodes are ready to be added back to the cluster as functional members.  Start-DatabaseAvailabilityGroup is used to bring the nodes back to the cluster.

 

Occasionally when running Start-DatabaseAvailabilityGroup the following error is thrown:

 

WARNING: Server 'DAG-4' failed to be started as a member of database availability group 'DAG'. Error: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API’ "AddClusterNode() (MaxPercentage=37) failed with 0x13af. Error: The cluster node is not valid"' failed. [Server: DAG-1.domain.com]

 

D:UtilitiesERR>err 0x13af
# for hex 0x13af / decimal 5039
  ERROR_CLUSTER_INVALID_NODE                                     winerror.h
# The cluster node is not valid.
  SQL_5039_severity_16                                           sql_err
# MODIFY FILE failed. Specified size is less than current
# size.
# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x13af
# for hex 0x13af / decimal 5039
  ERROR_CLUSTER_INVALID_NODE                                     winerror.h
# The cluster node is not valid.
# 3 matches found for "0x13af"

 

Why does this error occur?

 

Most customers will never see this error, and those that do typically see this error only in testing, where the commands associated with the Datacenter Switchover process are executed in a quick sequential fashion.  When Start-DatabaseAvailabilityGroup is run the command steps through the stopped mailbox servers list.  If you look at the list carefully you will notice that the nodes most likely appear in reverse order that then they were originally added to the DAG.

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers,startedmailboxservers,stoppedmailboxservers

Name                  : DAG
Servers               : {MBX-4, MBX-3, MBX-2, MBX-1}
StartedMailboxServers : {MBX-3.domain.com, MBX-4.domain.com}
StoppedMailboxServers : {MBX-2.domain.com, MBX-1.domain.com}

In this example, MBX-1 was added before MBX-2.  This would indicate that MBX-1 was assigned Node ID 1 and MBX-2 was assigned Node ID 2.  As nodes are added to a cluster, the first available node ID is recycled. In this example, Start-DatabaseAvailabilityGroup first detects the server MBX-2, therefore this server is the first server attempted to add to the existing cluster.  The cluster service then attempts to assign Node ID 1 to MBX-2.  For a short period of time the cluster service caches the previous Node ID to Server mapping that originally existed.  Therefore, in cache, Node ID 1 is still assigned to server MBX-1.  In this instance though we are trying to add server MBX-2 and the cluster by default attempts to assign Node ID 1 – resulting in a collision between the desired Node ID assignment and what exists in cache.  This ultimately results in the invalid node ID error returned to the cmdlet.

 

In the case of using the ADsite parameter with Start-DatabaseAvailabilityGroup, adding MBX-2 fails while adding server MBX-1 succeeds.  Based on the previous explanation this is expected, the cmdlet failed to add MBX-2 and now moved to MBX-1.  MBX-1 during the join process is assigned the first available node ID, in this case Node ID 1, which matches what previously existed in cache.  Since this match existed, the join was successful.  Executing the command multiple times with the –ADSite parameter will eventually result in all nodes being successfully added to the DAG.  If you run Start-DatabaseAvailabilityGroup with the –mailboxServer parameter, the error will continue to be seen in the short term unless the server specified matches the first available node ID.

 

Again, most administrators will not encounter this error during production site activations.  The time between when Restore-DatabaseAvailablityGroup is run and the time that the primary site is available to be added back to the DAG is sufficient for the cache to expire.  In some circumstances it may become necessary to restart the Cluster service on the surviving nodes to force the cache to expire and correct this error.  You can also use Start-DatabaseAvailabilityGroup with the –mailboxServer parameter to add the nodes back in the original order they were installed.

Outlook Web App offers the ability for users to add hyper links to certain objects within a message.  Commonly this feature is utilize when converting a text statement to a particular URL.  To invoke this feature the message author would highlight a section of text and select the Insert HyperLink   action on the toolbar.

 

 

This option may not be visible depending on the organization of the reading pane in OWA – you may need to press the menu expansion button   to expose the Insert HyperLink button. 

 

Message authors may desire to insert a link to an embedded image within a message.  In this example an image was inserted in the message. 

 

 

The embedded image was selected to allow a hyperlink to be inserted.  In this instance the insert hyperlink button is unavailable.

 

 

The insert hyperlink button is not available for embedded images.  Does that mean that an embedded image cannot have a hyperlink?  No – using keyboard shortcuts message authors can add a hyperlink to an embedded message.  Selecting the picture the message author can utilizing the shortcut CRTL + L.  Using this keyboard shortcut brings up the add hyperlink dialog.

 

 

A standard URL can be entered into this dialog and OK pressed to commit the hyperlink to the image.  When the message is sent to the recipient the hyperlinked image is included.  Clicking the image will allow the message recipient to go to the destination website specified.

 

In a future revision of Office 365 the ability to utilize the add hyperlink button will be expanded to embedded images with Outlook Web App.

Customers may have users with a security principal within their domain that do not have a mailbox hosted within their messaging solution.  In this event customers choose to mail enable the user accounts.  In an Exchange environment mail enabled users accounts appear in the address list as if a local mailbox exists but mail destined for these objects are sent to the external email address stamped on the object.

 

When using directory synchronization with Office 365 these objects are subsequently replicated into the MSOL directory as user accounts.  Exchange online then detects the presence of these mail enabled user accounts and subsequently creates a mail enabled user object in the Exchange Online directory.  This is what allows the user to appear in the Exchange Online address list.

 

Let’s take a look at an example.

 

In the on premises active directory a user account is provisioned.  This account is what allows for logon privileges to the domain.

 

[PS] C:>Get-User BlogTest

Name                                                        RecipientType
—-                                                        ————-
Blog Test                                                   User

 

The account is then mail enabled using the enable-mailUser commandlet.

 

[PS] C:>Enable-MailUser BlogTest -ExternalEmailAddress user@microsoft.com

Name                                     RecipientType
—-                                     ————-
Blog Test                                MailUser

 

Using the get-mailUser commandlet we can validate the settings of this user including the external email address set on the object.

 

[PS] C:>Get-MailUser BlogTest | fl externalemailaddress,userprincipalname,displayname,emailaddresses,primarysmtpaddress
,recipienttype

ExternalEmailAddress : SMTP:user@microsoft.com
UserPrincipalName    : BlogTest@domain.com
DisplayName          : Blog Test
EmailAddresses       : {smtp:BlogTest@domain1.domain2.com, smtp:BlogTest@serviceDomain.mail.onmicrosoft.com,
                       smtp:BlogTest@domain.com, SMTP:user@microsoft.com}
PrimarySmtpAddress   : user@microsoft.com
RecipientType        : MailUser

 

When directory synchronization occurs the object can be validated in the MSOL directory using get-MSOLUser.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@Domain2 | fl DisplayName,ProxyAddresses

DisplayName    : Blog Test
ProxyAddresses : {smtp:BlogTest@ServiceDomain.onmicrosoft.com, smtp:BlogTest@Domain2,
                 smtp:BlogTest@DOMAIN1.Domain2, smtp:BlogTest@ServiceDomain.mail.onmicrosoft.com…}

 

When the provisioning process has had time to detect and operate on the object a mail user object is created in Exchange Online.  This can be validated with the get-MailUser command when connected to Exchange Online.

 

PS C:> Get-MailUser BlogTest | fl externalemailaddress,userprincipalname,displayname,emailaddresses,primarysmtpaddress,
recipientType

ExternalEmailAddress : SMTP:user@microsoft.com
UserPrincipalName    : BlogTest@domain.com
DisplayName          : Blog Test
EmailAddresses       : {SMTP:user@microsoft.com, smtp:BlogTest@servicedomain.mail.onmicrosoft.com,
                       smtp:BlogTest@domain1.domain2.com, smtp:BlogTest@domain.com…}
PrimarySmtpAddress   : user@microsoft.com
RecipientType        : MailUser

 

At this time the object is fully provisioned and will appear in the Exchange Online global address list.  Users who select this object from the global address list will successfully send email to the remote mailbox.

 

 

 

There may exist at some time a reason to remove the on premises Active Directory user account.  When this object is removed and directory synchronization is performed this should result in the removal of the MSOL user account as well as the mail user account within Exchange Online.  Let’s look at an example.

 

Using Active Directory Users and Computers the on premises AD object is deleted.  This can be validated with Get-User.

 

[PS] C:>Get-User BlogTest

The operation couldn't be performed because object 'BlogTest' couldn't be found on 'Server1.domain.domain.com'.
    + CategoryInfo          : NotSpecified: (:) [Get-User], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=MAIL,RequestId=95b95c5e-03c2-406a-9400-c490c0d7cddb,TimeStamp=5/13/2014 1:34:34PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 4549B0D2,Microsoft.Exchange.Management.RecipientTasks.GetUser
    + PSComputerName        : mail.domain.domain.com

 

This also results in the removal of the on premises mail user object.  This can be validated with Get-MailUser.

 

[PS] C:>Get-MailUser BlogTest

The operation couldn't be performed because object 'BlogTest' couldn't be found on 'Server1.domain1.domain2.com'.
    + CategoryInfo          : NotSpecified: (:) [Get-MailUser], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=MAIL,RequestId=0106f632-ad95-4f07-b054-4e749e437cfc,TimeStamp=5/13/2014 1:35:55PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 84EC86F5,Microsoft.Exchange.Management.RecipientTasks.GetMailUser
    + PSComputerName        : mail.domain1.domain2.com

 

When directory synchronization has completed we can validate the object no longer exists in the MSOL directory using get-MSOLUser.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@domain2.com | fl DisplayName,ProxyAddresses

Get-MsolUser : User Not Found.  User: BlogTest@domain2.com.
At line:1 char:1
+ Get-MsolUser -UserPrincipalName BlogTest@domain2.com | fl DisplayName,Pro …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [Get-MsolUser], MicrosoftOnlineException
    + FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UserNotFoundException,Microsoft.Online.Administration.Automation.GetUser

 

When the provisioning process has detected the deletion the mail user object is removed from Exchange Online.  This can be validated with get-MailUser.

 

PS C:> Get-MailUser BlogTest

The operation couldn't be performed because object 'BlogTest' couldn't be found on 'CO1PR06A002DC02.NAMPR06A002.prod.outlook.com'.
    + CategoryInfo          : NotSpecified: (:) [Get-MailUser], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=BN1PR06MB101,RequestId=c3cde9d7-e638-4808-8891-65d539689698,TimeStamp=5/13/2014 1:40:19 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 782EAA9B,Microsoft.Exchange.Management.RecipientTasks.GetMailUser
    + PSComputerName        : pod51043psh.outlook.com

 

The object after deletion from the Exchange Online directory will no longer appear in the address list. 

 

 

End users who desire to email this object would have to address the message manually to the external email address.  For example:

 

 

This is where the issue arises.  In this instance the user has successfully addressed an email to the external email address but the mail is returned with a non-delivery report.

 

 

Delivery has failed to these recipients or groups:

Blog Test

The email address you entered couldn't be found. Check the recipient's email address then try to resend the message. For more tips to resolve this issue see DSN code 5.1.1 in Exchange Online. If the problem continues contact your help desk.

 

The diagnostic information states the following:

 

Diagnostic information for administrators:
Generating server: DM2PR0601MB0953.namprd06.prod.outlook.com
IMCEAEX-_o=ExchangeLabs_ou=Exchange+20Administrative+20Group+20+28FYDIBOHF23SPDLT+29_cn=Recipients_cn=0729272369574e7d945aeeecf1afd94c-Blog+20Test@namprd06.prod.outlook.com
Remote Server returned '550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found'
Original message headers:

Received: from DM2PR0601MB0953.namprd06.prod.outlook.com (25.160.25.145) by
DM2PR0601MB0953.namprd06.prod.outlook.com (25.160.25.145) with Microsoft SMTP
Server (TLS) id 15.0.944.11; Tue, 13 May 2014 13:43:53 +0000
Received: from DM2PR0601MB0953.namprd06.prod.outlook.com ([25.160.25.145]) by
DM2PR0601MB0953.namprd06.prod.outlook.com ([25.160.25.145]) with mapi id
15.00.0944.000; Tue, 13 May 2014 13:43:53 +0000
Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: Dev User <DevUser@servicedomain.onmicrosoft.com>
To: Blog Test <user@microsoft.com>
Subject: Test Message
Thread-Topic: Test Message
Thread-Index: AQHPbrFT/8JoaY496EePsUIz8dp/Tw==
Date: Tue, 13 May 2014 13:43:52 +0000
Message-ID: <1399988632393.9363@FortMillRescueSquad.onmicrosoft.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator: <1399988632393.9363@servicedomain.onmicrosoft.com>
MIME-Version: 1.0
X-Originating-IP: [167.220.151.116]
Return-Path: DevUser@serviceDomain.onmicrosoft.com

 

For many administrators this NDR is recognizable.  The same NDR is generated when an object is deleted from the environment but the email is addressed using nickname or recipient cache.  In this instance though the email was not addressed using nickname or recipient cache but rather the SMTP address was fully typed in the TO: line.  (Note:  The email address was manually removed from the nickname cache prior to addressing the email to ensure that automatic resolution does not occur.)

 

Why is the mail NDRing.  When a user account is removed from the MSOL directory the account is placed into a soft deleted state.  This can be validated with get-MSOLUser –returnedDeletedUsers.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@domain.com -ReturnDeletedUsers

UserPrincipalName                       DisplayName                             isLicensed
—————–                       ———–                             ———-
BlogTest@domain.com                     Blog Test                               False

 

While the user remains in a soft deleted state within the MSOL directory the corresponding mail user object remains in a soft deleted state within Exchange Online.  It is believed that the recipient resolvers within transport detect the presences of the soft deleted mail user object within the Exchange Online directory and subsequently produce a non-delivery report based on the state of this object.

 

This mail is legitimate though – how can the issue be resolved?

 

In order to remove the soft deleted mail user object from the Exchange Online directory the soft deleted user object must be removed from the MSOL directory.  This can be accomplished with remove-MSOLUser.

 

PS C:> Remove-MsolUser -UserPrincipalName BlogTest@domain.com -RemoveFromRecycleBin

Confirm
Continue with this operation?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): y

 

The results can be validated with Get-MSOLUser.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@domain.com -ReturnDeletedUsers
Get-MsolUser : User Not Found.  User: BlogTest@fortmillrescue.com.
At line:1 char:1
+ Get-MsolUser -UserPrincipalName BlogTest@fortmillrescue.com -ReturnDeletedUsers
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [Get-MsolUser], MicrosoftOnlineException
    + FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UserNotFoundException,Microsoft.Online.Administration.Automation.GetUser

 

The provisioning process will eventually detect the deletion of the object from the MSOL directory.  This will cause the soft deleted mail user object to be purged from the Exchange Online directory.  After this object is purged mail should deliver successfully to the external account when the full external email address is utilized.

 

 

At this time anytime a mail user object is removed from the on premises directory the corresponding soft deleted MSOL object would need to also be removed in order to ensure that mail addressed to the external object will immediately function.  In many cases the soft deleted MSOL object will expire, resulting in the removal of the soft deleted mail user object from the Exchange Online directory, prior to this issue being noticed by the end user population.

 

Our development teams are aware of this behavior and are considering potential future service modifications.

In both Exchange 2010 and Exchange 2013, customers can deploy one or more replication networks in a database availability group (DAG).  There can be many reasons for using a replication network, but in most cases they are used to provide a dedicated log shipping channel between members of the same DAG.

 

As documented on TechNet, when a replication network fails, replication should automatically failover to the DAG’s MAPI network:

· http://technet.microsoft.com/en-us/library/dd638104(v=exchg.150).aspx (Exchange 2013)

· http://technet.microsoft.com/en-us/library/dd638104(v=exchg.141).aspx (Exchange 2010)

 

“In the event of a failure affecting the Replication network, if the MAPI network is unaffected by the failure, log shipping and seeding operations will revert to use the MAPI network, even if the MAPI network has it's ReplicationEnabled property set to False.”

 

Log shipping operations occur by connecting to the Microsoft Exchange Replication service on the server that hosts the active database copy, on TCP port 64327 using a random ephemeral port on the passive server.  Log files are then pushed from the active server to the passive server via this channel.  There can exist several issues that result in this log shipping channel being interrupted.  For example, a firewall may block port 64327, static routes may be missing on multi-subnet replication networks, or an intermediary network device may not route traffic correctly. 

 

In the following example, we have a 4-member DAG running Exchange 2013.  There is a single active database that is replicated to three other servers.

 

 

The DAG has two networks that each has two subnets.  These networks represent the MAPI and Replication networks for the DAG.  Automatic network detection in Exchange 2013 was disabled for this example.

 

RunspaceId         : 60e6ae0f-e69d-4ae6-9fcb-8c99ea9fd21f
Name               : MapiDagNetwork
Description        :
Subnets            : {{192.168.0.0/24,Up}, {192.168.1.0/24,Up}}
Interfaces         : {{MBX-1,Up,192.168.0.11}, {MBX-2,Up,192.168.0.12}, {MBX-3,Up,192.168.1.11},
                     {MBX-4,Up,192.168.1.12}}
MapiAccessEnabled  : True
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAGMapiDagNetwork
IsValid            : True
ObjectState        : New

RunspaceId         : 60e6ae0f-e69d-4ae6-9fcb-8c99ea9fd21f
Name               : ReplicationDagNetwork01
Description        :
Subnets            : {{10.0.1.0/24,Up}, {10.0.0.0/24,Up}}
Interfaces         : {{MBX-1,Up,10.0.0.1}, {MBX-2,Up,10.0.0.2}, {MBX-3,Up,10.0.1.1}, {MBX-4,Up,10.0.1.2}}
MapiAccessEnabled  : False
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAGReplicationDagNetwork01
IsValid            : True
ObjectState        : New

 

Using Get-MailboxDatabaseCopyStatus with the –ConnectionStatus switch, we can verify that the Replication network is currently in use:

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -connectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

 

The router servicing the network link between the two subnets of the Replication network is shutdown.  This will block replication from succeeding over the Replication network.  What happens to the database copies?

 

[PS] C:>Get-MailboxDatabaseCopyStatus * | fl name,status

Name   : DAG-DB0MBX-1
Status : Mounted

Name   : DAG-DB0MBX-2
Status : Healthy

Name   : DAG-DB0MBX-3
Status : DisconnectedAndHealthy

Name   : DAG-DB0MBX-4
Status : DisconnectedAndHealthy

 

In this example, the database copies enter a DisconnectedAndHealthy state.  Reviewing the connection status, we note that the connection has timed out between the servers; this is expected since the route is down.

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {{MBX-2,ReplicationDagNetwork01}}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

 

Why didn’t the Replication network failover to the MAPI network in this case?  For one thing, the ability to establish a log shipping channel is not one of the criteria that the Replication service uses to determine the health of a given network.  The Replication service relies on feedback from the Cluster service regarding individual network interface status in order to determine the health of a log shipping channel.

 

For each subnet that exists on a DAG member, an associated cluster network is created.  In this example, there are 4 subnets and therefore there are 4 cluster networks.

 

[PS] C:>Get-ClusterNetwork | fl

Name  : Cluster Network 1
State : Up

Name  : Cluster Network 2
State : Up

Name  : Cluster Network 3
State : Up

Name  : Cluster Network 4
State : Up

 

 

The interface associated with each of these subnets is included in the appropriate cluster network.  Each of these interfaces has a status based on status reporting in Windows Failover Clustering.

 

[PS] C:>Get-ClusterNetworkInterface | fl

Name    : MBX-1 – LAN-A
Node    : MBX-1
Network : Cluster Network 1
State   : Up

Name    : MBX-2 – LAN-A
Node    : MBX-2
Network : Cluster Network 1
State   : Up

Name    : MBX-1 – REPL-A
Node    : MBX-1
Network : Cluster Network 2
State   : Up

Name    : MBX-2 – REPL-A
Node    : MBX-2
Network : Cluster Network 2
State   : Up

Name    : MBX-3 – REPL-B
Node    : MBX-3
Network : Cluster Network 3
State   : Up

Name    : MBX-4 – REPL-B
Node    : MBX-4
Network : Cluster Network 3
State   : Up

Name    : MBX-3 – LAN-B
Node    : MBX-3
Network : Cluster Network 4
State   : Up

Name    : MBX-4 – LAN-B
Node    : MBX-4
Network : Cluster Network 4
State   : Up

 

 

In this scenario, the Cluster service considers all of the interfaces as “UP.”  Since the interfaces are UP, the Replication service does not failover to the MAPI network even though replication cannot occur over the replication network. 

 

If the Cluster service reports a network as “FAILED” the behavior is different.  On a server hosting a passive database copy, the network cable was removed from the Replication network interface, causing the cluster to mark that interfaces as failed:

 

[PS] C:>Get-ClusterNetworkInterface | fl

Name    : MBX-1 – LAN-A
Node    : MBX-1
Network : Cluster Network 1
State   : Up

Name    : MBX-2 – LAN-A
Node    : MBX-2
Network : Cluster Network 1
State   : Up

Name    : MBX-1 – REPL-A
Node    : MBX-1
Network : Cluster Network 2
State   : Up

Name    : MBX-2 – REPL-A
Node    : MBX-2
Network : Cluster Network 2
State   : Up

Name    : MBX-3 – REPL-B
Node    : MBX-3
Network : Cluster Network 3
State   : Up

Name    : MBX-4 – REPL-B
Node    : MBX-4
Network : Cluster Network 3
State   : Failed

Name    : MBX-3 – LAN-B
Node    : MBX-3
Network : Cluster Network 4
State   : Up

Name    : MBX-4 – LAN-B
Node    : MBX-4
Network : Cluster Network 4
State   : Up

 

The copy status for the databases hosted on MBX-4 is healthy.

 

[PS] C:>Get-MailboxDatabaseCopyStatus *

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DAG-DB0MBX-1                                 Mounted         0         0                                  Healthy
DAG-DB0MBX-2                                 Healthy         0         0           4/29/2014 2:11:47 PM   Healthy
DAG-DB0MBX-3                                 Healthy         0         0           4/29/2014 2:11:47 PM   Healthy
DAG-DB0MBX-4                                 Healthy         0         0           4/29/2014 2:11:47 PM   Healthy

 

When reviewing the connection status for the databases on MBX-4, we see that the Replication service is using the MAPI network for log shipping.  Servers that have no issues with the Replication interface continue to use that interface.

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,MapiDagNetwork}
OutgoingConnections       :

 

When the interface was marked as failed, the Replication service successfully failed over to the MAPI network.

 

But what happens if the interface is not FAILED, but the interface configuration is invalid?   In this event, the Cluster service cannot pass cluster traffic between these two interfaces and the associated network is marked as partitioned.

 

[PS] C:>Get-ClusterNetwork | fl

Name  : Cluster Network 1
State : Up

Name  : Cluster Network 2
State : Up

Name  : Cluster Network 3
State : Partitioned

Name  : Cluster Network 4
State : Up

 

The interfaces within the partitioned network are marked as unreachable.

 

[PS] C:>Get-ClusterNetworkInterface | fl

Name    : MBX-1 – LAN-A
Node    : MBX-1
Network : Cluster Network 1
State   : Up

Name    : MBX-2 – LAN-A
Node    : MBX-2
Network : Cluster Network 1
State   : Up

Name    : MBX-1 – REPL-A
Node    : MBX-1
Network : Cluster Network 2
State   : Up

Name    : MBX-2 – REPL-A
Node    : MBX-2
Network : Cluster Network 2
State   : Up

Name    : MBX-3 – REPL-B
Node    : MBX-3
Network : Cluster Network 3
State   : Unreachable

Name    : MBX-4 – REPL-B
Node    : MBX-4
Network : Cluster Network 3
State   : Unreachable

Name    : MBX-3 – LAN-B
Node    : MBX-3
Network : Cluster Network 4
State   : Up

Name    : MBX-4 – LAN-B
Node    : MBX-4
Network : Cluster Network 4
State   : Up

 

 

What happens to the copies that are hosted on the servers with an unreachable interface status?

 

[PS] C:>Get-MailboxDatabaseCopyStatus * | fl name,status

Name   : DAG-DB0MBX-1
Status : Mounted

Name   : DAG-DB0MBX-2
Status : Healthy

Name   : DAG-DB0MBX-3
Status : DisconnectedAndHealthy

Name   : DAG-DB0MBX-4
Status : DisconnectedAndHealthy

 

An unreachable interface is not the same as a failed interface.  This resulted in the databases entering a disconnected state and replication not failing over to the MAPI network.  The connection status confirms this failure.

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

 

The network failure detection mechanism is the same in both Exchange 2010 and Exchange 2013, as well as the same in Windows 2008 R2, Windows 2012, and Windows 2012 R2 failover clusters.  In order for the Replication service to detect a failure of a Replication network, the operating system and Cluster service must report the underlying interface as failed.  If the Cluster service reports any other status for the interface, the Replication service will consider the network to be valid and replication will not failover to another network.

In Office 365 mailboxes are migrated to the service by configuring a migration batch.  When establishing the move configuration settings a “More Options” button was present.  The “More Options” button allowed administrators to configure the bad item limit and large item limit when the migration batch is created.

 

 

In current versions of the Office 365 / Exchange management tools the “More Options” button is missing.

 

 

The “More Options” button was removed as a part of the Office 365 / Exchange Management tools upgrade. 

 

If the bad item limit and large item limit must be adjusted, they must now be adjusted after the migration batch has been created.  These settings can only be adjusted on a migration batch that is not in progress. 

 

When creating new migration batches where these settings should be modified administrators should select the option “Manually start the batch later (by selecting it in the migration dashboard and then clicking Start)”.  This will stage the migration batch but not start it, allowing administrators to immediately adjust the bad item and large item limits.  The status of the migration batch will be “created”.

 

 

If the migration batch was created and has started it must be stopped.  This can be performed by selecting the migration batch and pressing the stop button (square button).  This will place the migration batch into a “stopped” state.

 

 

With the migration batch created or stopped the properties of the batch can be accessed by selecting the batch and pressing the edit button (pencil button).  On the properties of the batch, on the general tab, the bad item limit and large item limit are editable. 

 

 

After entering the appropriate values the save button can be utilized to commit the changes to the batch.  The batch can then be started or resumed by selecting the batch and pressing the resume button (play button). 

 

At this time we expect the “More Options” button to appear in the Office 365 / Exchange management tools in a future version. 

I recently worked with a customer that was utilizing directory synchronization to create distribution groups in their on-premises active directory and have them mail enabled within Office 365.  They noticed that when creating these distribution groups the display name in Office 365 was automatically changed and differed from the display name set on-premises.

 

Let’s take a look…

 

Using Active Directory Users and Computers a new distribution list is created.

 

 

Using the attribute editor the following attributes were manually set:

 

DisplayName:  TestDL

ProxyAddresses:  SMTP:TestDL@domain.com

Mail:  TestDL@Domain.com

 

After the distribution list was created directory synchronization was performed.  Monitoring directory synchronization we can validate the properties of the group synchronized into Office 365.

 

 

Using Powershell connected to Office 365 we can validate the properties of the MSOL Group.

 

PS C:Userstimmcmic> Get-MsolGroup -ObjectId df617972-c874-4d6d-ba73-993e73219514 | fl

ExtensionData    : System.Runtime.Serialization.ExtensionDataObject
CommonName       : TestDL
Description      :
DisplayName      : TestDL
EmailAddress     : TestDL@domain.com
Errors           : {}
GroupType        : DistributionList
IsSystem         : False
LastDirSyncTime  : 4/9/2014 2:56:34 PM
ManagedBy        :
ObjectId         : df617972-c874-4d6d-ba73-993e73219514
ProxyAddresses   : {smtp:TestDL@domain.onmicrosoft.com, SMTP:TestDL@domain.com}
ValidationStatus : Healthy

 

The attributes of the group within Office 365 and the on-premises Active Directory match after directory synchronization has completed.

 

When reviewing the properties of the same distribution list from Office 365 Exchange it is noted that the name has changed.  In this case a prefix was appended DL_:

 

PS C:Userstimmcmic> Get-DistributionGroup TestDL | fl NAME,DISPLAYNAME,EMAILADDRESSES

Name           : DL_TestDL
DisplayName    : DL_TestDL
EmailAddresses : {SMTP:TestDL@domain.com, smtp:TestDL@domain.onmicrosoft.com}

Where did the DL_ come from?  Exchange and Office 365 allow administrators to define a distribution group naming policy.  The distribution group naming policy is set in the organization’s configuration.  More information on distribution group naming policies can be found here:  https://aka.ms/Bry5y2

 

In this instance the tenant had a distribution group naming policy that utilized DL_<GroupName>:

 

PS C:Userstimmcmic> Get-OrganizationConfig | fl distributiongroupnamingpolicy

DistributionGroupNamingPolicy : DL_<GroupName>

 

When a distribution group naming policy is utilized after directory synchronization has successfully synchronized the group and the new group is provisioned within Exchange Office 365 the naming policy will be applied.  The MSOL object will continue to reflect the attributes as synchronized from the on-premises Active Directory but the Exchange object will reflect the attributes modified by the naming policy.

 

In our case it was desired that the naming policy would not be applied.  This required removing the naming policy from the organizational configuration:

 

PS C:Userstimmcmic> Set-OrganizationConfig -DistributionGroupNamingPolicy:$NULL

 

By removing the naming policy the Exchange Office 365 groups will now reflect the attributes as synchronized from the on-premises Active Directory.

Recently I was re-reading a blog post from 2012 about on Tuning Failover Cluster Network Thresholds that was posted by Elden Christensen, a Principal PM on the Windows Failover Cluster team. I think Elden’s post is a must-read for anyone planning a highly-available Exchange deployment.  One of the reasons I find this post an excellent read is that it addresses many things administrators need to understand when they decide to tune cluster heartbeat subnet delays and thresholds in Exchange environments.

 

The post first makes an excellent point in addressing that the changes to these thresholds alter the amount of time it takes to detect that a node is down.  Elden uses a great metaphor here:

 

“Think of it like your cell phone, when the other end goes silent how long are you willing to sit there going “Hello?… Hello?… Hello?” before you hang-up the phone and call the person back.  When the other end goes silent, you don’t know when or even if they will come back.”

As subnet thresholds are adjusted up, this increases the amount of time it takes to detect a failure. The higher the values, the longer it takes to detect a failure, and therefore the longer it takes to act on that failure. There is a balance between reacting quickly to a failure and providing resiliency to transient networking issues.

 

The other point that I think is worth understanding is the number of times these values are adjusted in the absence of an analysis or correction of underlying networking issues.  Elden sums this up, too, and I could not agree with him more:

 

“It critical to recognize that cranking up the thresholds to high values does not fix nor resolve the transient network issue, it simply masks the problem by making health monitoring less sensitive. The #1 mistake made broadly by customers is the perception of not triggering cluster health detection means the issue is resolved (which is not true!). I like to think of it, that just because you choose not to go to the doctor it does not mean you are healthy. In other words, the lack of someone telling you that you have a problem does not mean the problem went away.”

I often find myself in conversations with customers who have changed these values and have the perception that something is “fixed.” There are legitimate cases where these values need to be changed – but I always encourage a networking analysis enables you to understand what issues you are facing and how adjusting these values would help. Unfortunately, it seems that adjusting these thresholds without this understanding is far more common than it should be.

 

I strongly encourage all Exchange administrators to read Elden’s post.

When accounts are provisioned in Office 365, there are certain self-service features that end users have access to without administrator interaction. For example, when logging in and selecting Office 365 settings, provisioned accounts have access to software downloads and the change password feature.

 

 

 

When an account has an Exchange license added to it, the provisioning process creates a mailbox for it in the service. This automatically enables Outlook Web App (OWA) for the user. Depending on the license type and the age of the mailbox, the user logon experience may be different.  For example, on new accounts when the user logs in they may be taken to the introduction portal page.

 

 

For accounts that are aged or possess an Exchange-only license type the Introduction page may be automatically skipped and the user immediately redirected to OWA.

 

 

When a user is redirected to OWA they can access their self-service options by selecting the gear in the upper right-hand corner and selecting Office 365 Settings.

 

 

One of the administrative features we provide is the ability to disable OWA. This allows the administrator to restrict mailbox access via OWA, while allowing access from other clients – for example, Outlook. When OWA is enabled the feature shows enabled with the Disable action available.

 

 

When selecting the disabled option OWA will be unavailable. The mailbox then shows a setting of disabled with the Enable action available.

 

 

When a user attempts to access OWA from the Introduction page (selecting Outlook in the action bar) they are prompted with the following error:

X-OWA-Error: Microsoft.Exchange.Data.Storage.AccountDisabled.Exception:

 

 

When a user has landed on this page and they require access to other services, they must manually return to the Introduction page. The issue arises when the mailbox is aged or has been provisioned with an Exchange-only license as the Introduction page is bypassed and the user is immediately directed to OWA. The user cannot use the back feature in the browser to return to the Introduction page and no option to select Office 365 settings is present on the failure page.

 

When this situation is present users can be directed to the following URL.  This URL will bypass the automatic OWA redirection and allow the initial page presented to be the user. 

 

https://portal.microsoftonline.com/IWGetStarted15.aspx?DisableIWLanding=true

 

 

This URL should allow administrators to continue to provide access to self-service options while OWA is in a disabled state.

In some circumstances it becomes necessary to migrate users quickly between servers within Database Availability Groups.  In some instances move mailbox is not an option.  When storage can be maintained we can utilize the swing method to complete the migration.

 

Some instances where these instructions have been implemented include:

 

• Replacement of off lease hardware (for example nodes) where additional storage does not exist for move mailbox or to replicate database copies.
• Upgrading nodes to new operating systems by migrating storage / databases from a previous version of windows and Exchange to a new version of Windows with same Exchange version.

 

The swing method involves identifying a database copy to be moved between nodes, migrating the storage between nodes, and then migrating additional copies of the database between nodes. 

 

There are several considerations when deciding to implement the storage swing migration.  Some of these considerations include:

 

• The complexity of the steps and the ability to test prior to implementing against production users.
• The loss of all lagged database copies.
• The need for end user downtime to complete the transition.
• Maintaining enough storage to hold all log files during the transition process due to log file truncation being blocked either by circular logging or backups.
• Content indexes must be completely rebuilt once databases and storage are migrated.

 

In this article we have implemented the following architecture for testing and documentation:

 

 

Step 0:  Ensure all support teams are aware of the actions to be performed.

 

It is important that all support teams are prepared for the actions to be taken in these steps.  Ensuring that storage can be migrated quickly is paramount to reducing downtime.  Also depending on your hardware, additional steps may need to be performed to ensure that storage can be imported.  For example, in DAS environments when moving storage chassis between nodes, RAID configurations must be imported into new controllers (as opposed to SAN environments, where LUNs can be mapped to different servers).

 

Step 1:  Ensure storage has appropriate labels.

When creating partitions, you can assign labels. Maintaining meaningful labels on the storage helps you to be aware of the volumes you are working with. You cannot rely on the disk numbers because as storage is migrated between servers, different disk numbers may be assigned. But when disks are moved between servers, disk labels are maintained.

 

In this example, I have generic volume names on my server – for example Data0 / Data1 / Data2…etc. 

 

 

These volume names are generic and do not help identify the data contained within these volumes.  Using the Disk Management tool, I can change the volume names to something more meaningful, such as an indication of the data being stored.

 

 

 

Naming volumes in this manner can reduce potential confusion when migrating storage between servers.

 

Step 2:  Create new database objects on the new database availability groups for the databases that will be migrated.

 

In this step, you create new database objects on the target DAG. I recommend that these new databases be created on the same DAG member.  This will serve as the node where the initial storage migration will occur and where you should be able to quickly restore services.  In larger DAGs, where there is no single node that will host all database copies, you can spread the new databases out across the DAG members.  Ensure to keep track of these database locations so storage is migrated to the appropriate servers.

 

In the example below, I create the new mailbox databases, but I don’t specify a path for log files or databases.  Thus, these databases are created at the default location (%ProgramFiles%MicrosoftExchange Serverv14Mailbox). The databases are not mounted at this point, and therefore no storage is being used. I am creating the databases and then allowing sufficient time for Active Directory replication and for the Microsoft Exchange Replication and Information Store services to detect the databases.

 

[PS] C:>New-MailboxDatabase -Name NEW-DB0 -Server MBX-2A

Name                           Server          Recovery        ReplicationType
—-                           ——          ——–        —————
NEW-DB0                        MBX-2A          False           None

[PS] C:>New-MailboxDatabase -Name NEW-DB1 -Server MBX-2A

Name                           Server          Recovery        ReplicationType
—-                           ——          ——–        —————
NEW-DB1                        MBX-2A          False           None

I can validate the configuration with the following command:

 

[PS] C:>Get-MailboxDatabase -Server MBX-2A -Status | fl name,*mounted*,*path*

Name                    : NEW-DB0
MountedOnServer         : MBX-2A.exchange.msft
Mounted                 : False
EdbFilePath             : C:Program FilesMicrosoftExchange ServerV14MailboxNEW-DB0NEW-DB0.edb
LogFolderPath           : C:Program FilesMicrosoftExchange ServerV14MailboxNEW-DB0
TemporaryDataFolderPath :

Name                    : NEW-DB1
MountedOnServer         : MBX-2A.exchange.msft
Mounted                 : False
EdbFilePath             : C:Program FilesMicrosoftExchange ServerV14MailboxNEW-DB1NEW-DB1.edb
LogFolderPath           : C:Program FilesMicrosoftExchange ServerV14MailboxNEW-DB1
TemporaryDataFolderPath :

Step 3:  Remove all truncate and replay lags from database copies.

In this step, you disable truncation and replay lag settings from all database copies that have them configured.  It is necessary to have all databases up to date before migrating them to new storage.  By disabling replay and truncation lag settings, the administrator can decrease the amount of downtime required to move storage between nodes. Truncation and replay lag settings are dynamic, and when disabled, log file replay and log file truncation will start on lagged copies at the next Replication service update cycle.  Sufficient time should be allowed between this step and the day of migration in order to allow any lagged copies to replay outstanding log files.

 

In the following example, databases assigned to MBX-1C are databases with lagged copies:

 

[PS] C:>Get-MailboxDatabaseCopyStatus *

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DB0MBX1-A                                    Mounted         0         0                                  Healthy
DB1MBX1-A                                    Healthy         0         0           1/28/2014 1:59:43 PM   Healthy
DB1MBX-1B                                    Mounted         0         0                                  Healthy
DB0MBX-1B                                    Healthy         0         0           1/28/2014 1:36:57 PM   Healthy
DB0MBX-1C                                    Healthy         0         267         1/28/2014 1:36:57 PM   Healthy
DB1MBX-1C                                    Healthy         0         253         1/28/2014 1:59:43 PM   Healthy
NEW-DB0MBX-2A                                Dismounted      0         0                                  Unknown
NEW-DB1MBX-2A                                Dismounted      0         0                                  Unknown

 

To disable the lagged copy we utilize the set-mailboxdatabasecopy command:

 

set-mailboxdatabasecopy DB0MBX-1C –replayLagTime 0.0:0:0 –truncationLagTime 0.0:0:0

set-mailboxdatabasecopy DB1MBX-1C –replayLagTime 0.0:0:0 –truncationLagTime 0.0:0:0

Eventually, the queues should decrease down to zero.

 

[PS] C:>Get-MailboxDatabaseCopyStatus *

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DB0MBX1-A                                    Mounted         0         0                                  Healthy
DB1MBX1-A                                    Healthy         0         0           1/28/2014 1:59:43 PM   Healthy
DB1MBX-1B                                    Mounted         0         0                                  Healthy
DB0MBX-1B                                    Healthy         0         0           1/28/2014 1:36:57 PM   Healthy
DB0MBX-1C                                    Healthy         0         0           1/28/2014 1:36:57 PM   Healthy
DB1MBX-1C                                    Healthy         0         0           1/28/2014 1:59:43 PM   Healthy
NEW-DB0MBX-2A                                Dismounted      0         0                                  Unknown
NEW-DB1MBX-2A                                Dismounted      0         0                                  Unknown

Depending on the duration of the original lag settings, this procedure could take several hours or days to complete. Once the replay queues are at zero, the lagged copy is considered successfully disabled.

 

Step 4:  Validate database copy health

This step needs to be performed immediately before migrating storage.  It is imperative that all database copies be healthy prior to proceeding with further steps. You can use Get-MailboxDatabaseCopyStatus to validate that all database copies are healthy.

 

[PS] C:>Get-MailboxDatabaseCopyStatus *MBX1-A

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DB0MBX1-A                                    Mounted         0         0                                  Healthy
DB1MBX1-A                                    Healthy         0         0           1/29/2014 5:32:30 AM   Healthy

[PS] C:>Get-MailboxDatabaseCopyStatus *MBX-1B

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DB1MBX-1B                                    Mounted         0         0                                  Healthy
DB0MBX-1B                                    Healthy         0         0           1/29/2014 5:32:22 AM   Healthy

[PS] C:>Get-MailboxDatabaseCopyStatus *MBX-1C

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DB0MBX-1C                                    Healthy         0         0           1/29/2014 5:32:22 AM   Healthy
DB1MBX-1C                                    Healthy         0         0           1/29/2014 5:32:30 AM   Healthy

 

If any database is unhealthy, it needs to be fixed.  In some instances, that may require reseeding which can take several hours. Appropriate time should be allotted to ensure that any remediation steps can be followed.

 

Step 5:  Dismount databases and validate database dismount

Next, the databases that will be migrated are dismounted. They will need to remain dismounted until the storage has been successfully migrated, and services are restored in the new database availability group.

 

Use Dismount-Database to dismount the databases:

 

[PS] C:>Dismount-Database DB0 -Confirm:$False
[PS] C:>Dismount-Database DB1 -Confirm:$False

 

Use Get-MailboxDatabase –Status to verify the databases are dismounted.

 

[PS] C:>Get-MailboxDatabase DB0 -Status | fl *mounted*

MountedOnServer : MBX1-A.exchange.msft
Mounted         : False

[PS] C:>Get-MailboxDatabase DB1 -Status | fl *mounted*

MountedOnServer : MBX-1B.exchange.msft
Mounted         : False

 

Step 6:  Ensure log file copy and replay has completed.

 

There can be conditions that occur between validating copy status and dismounting databases that do not result in all log files being copied to the passive node. So in this step, you manually copy all log files to the passive node.

 

Using an administrative command prompt on the server hosting the passive copy, navigate to the log file directory for a database you are migrating.

 

Execute a command similar to the following – robocopy \<ActiveNode><Drive$><log folder path> . /E  (This example assumes the command prompt is already present in the target log file directory)

 

F:DB1>robocopy \mbx-1bf$DB1 . /e /xf *.chk

——————————————————————————
   ROBOCOPY     ::     Robust File Copy for Windows

——————————————————————————

  Started : Wednesday, January 29, 2014 9:50:13 AM
   Source : \mbx-1bf$DB1
     Dest : F:DB1

    Files : *.*

Exc Files : *.chk

  Options : *.* /S /E /DCOPY:DA /COPY:DAT /R:1000000 /W:30

——————————————————————————

                         364    \mbx-1bf$DB1
        *EXTRA Dir        -1    F:DB1incseedInspect
100%        New File               1.0 m        E01.log
                           0    \mbx-1bf$DB1IgnoredLogs
                           0    \mbx-1bf$DB1inspector

——————————————————————————

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         0         0         0         0         1
   Files :       364         1       363         0         0         0
   Bytes :  362.00 m    1.00 m  361.00 m         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00

   Speed :            22310127 Bytes/sec.
   Speed :            1276.595 MegaBytes/min.
   Ended : Wednesday, January 29, 2014 9:50:13 AM

 

This will ensure that any missing log files, as well as the updated ENN.log, are available on all copies for replay.

 

Step 7:  Replay all log files into databases.

Next, use eseutil to replay all log files into all database copies.  This will ensure that all copies are up to date prior to migrating storage to the remote nodes.  This step will be performed on all servers hosting a passive or active database copy.

 

Launch an administrative command prompt and navigate to the log file directory.

 

Run eseutil /r ENN where ENN is the first three digits of the log prefix for that log sequence.

 

F:DB1>eseutil /r e01

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.03
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating RECOVERY mode…
    Logfile base name: e01
            Log files: <current directory>
         System files: <current directory>

Performing soft recovery…
                      Restore Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
          ……………………………………………

Operation completed successfully in 21.938 seconds.

 

When completed on all database copies we can proceed to the verification step next.

 

Step 7:  Validate database headers.

 

Once all log files have been copied and replayed, you must ensure that all database copies reflect this work. Compare the database headers of each database to ensure that they are equal. Specifically, compare the attributes LastConsistent and LastDetached, which you can view using eseutil.

 

To dump the header of the database utilize eseutil /mh.

 

E:DB0>eseutil /mh DB0.edb

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.03
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode…
         Database: DB0.edb

DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x1c1c8028
  Actual Checksum: 0x1c1c8028

Fields:
        File Type: Database
         Checksum: 0x1c1c8028
   Format ulMagic: 0x89abcdef
   Engine ulMagic: 0x89abcdef
Format ulVersion: 0x620,17
Engine ulVersion: 0x620,17
Created ulVersion: 0x620,17
     DB Signature: Create time:01/27/2014 12:18:49 Rand:3626382 Computer:
         cbDbPage: 32768
           dbtime: 962549 (0xeaff5)
            State: Clean Shutdown
     Log Required: 0-0 (0x0-0x0)
    Log Committed: 0-0 (0x0-0x0)
   Log Recovering: 0 (0x0)
  GenMax Creation: 00/00/1900 00:00:00
         Shadowed: Yes
       Last Objid: 7494
     Scrub Dbtime: 0 (0x0)
       Scrub Date: 00/00/1900 00:00:00
     Repair Count: 0
      Repair Date: 00/00/1900 00:00:00
Old Repair Count: 0
  Last Consistent: (0x1E6,1,2CA)  01/29/2014 10:57:52
      Last Attach: (0x16A,1,270)  01/29/2014 10:12:49
      Last Detach: (0x1E6,1,2CA)  01/29/2014 10:57:52
             Dbid: 1
    Log Signature: Create time:01/27/2014 12:18:48 Rand:3652040 Computer:
       OS Version: (6.2.9200 SP 0 NLS ffffffff.ffffffff)

Previous Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Incremental Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Copy Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Previous Differential Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Full Backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

Current Shadow copy backup:
        Log Gen: 0-0 (0x0-0x0)
           Mark: (0x0,0,0)
           Mark: 00/00/1900 00:00:00

     cpgUpgrade55Format: 0
    cpgUpgradeFreePages: 0
cpgUpgradeSpaceMapPages: 0

       ECC Fix Success Count: none
   Old ECC Fix Success Count: none
         ECC Fix Error Count: none
     Old ECC Fix Error Count: none
    Bad Checksum Error Count: none
Old bad Checksum Error Count: none

  Last checksum finish Date: 00/00/1900 00:00:00
Current checksum start Date: 00/00/1900 00:00:00
      Current checksum page: 0

Operation completed successfully in 0.125 seconds.

 

After dumping the header of each database copy for the same database, compare the LastConsistent and LastDetach times.  If these times are equal across all copies of the same database, then log file copy and log file replay were successful.

 

DB0MBX-1A

Last Consistent: (0x1E6,1,2CA)  01/29/2014 10:57:52
Last Detach: (0x1E6,1,2CA)  01/29/2014 10:57:52

 

DB0MBX-1B

Last Consistent: (0x1E6,1,2CA)  01/29/2014 10:57:53
Last Detach: (0x1E6,1,2CA)  01/29/2014 10:57:53

 

DB0MBX-1C

Last Consistent: (0x1E6,1,2CA)  01/29/2014 10:57:52
Last Detach: (0x1E6,1,2CA)  01/29/2014 10:57:52

If any of the copies does not equal for any reason, the database should be mounted on the source server, and then start back at Step 4 of this document. If all database headers are equal, proceed with storage migration.

 

Step 8:  Migrate storage to the new node.

 

When moving storage to the new server, start by migrating from a server hosting the passive database copy. In this example, I will focus on DB0 which was passive on server MBX-1B. The steps to move storage between servers depend on your storage implementation, and as such they are not covered in this article.  These steps should have been tested and validated prior to this point.

 

I recommend migrating the storage from a single node first. This allows the original active database and storage to remain intact in case there are any issues with the storage migration or the database on the target server. After services have been established on the target server the additional databases and storage can be migrated.

 

In this example, the new databases were created on server MBX-2A. I am moving the storage from MBX-1B to MBX-2A.  After bringing the disks online on MBX-2A using the Disk Management tool, appropriate drive letters or mount points can be assigned.  IMPORTANT:  note the drive letters and paths used in this procedure.  You will need to repeat this step on other servers using the exact same paths. Failure to implement the same drive letters or paths will result in failure of subsequent steps.

 

Step 9:  Mount the migrated database on the new node.

In Step 2 above, you created your database objects. In this step, match one of those databases to the files that were moved from the original DAG. First, use Set-Mailbox to configure the allowFileRestoreFlag on each database.

 

Using set-mailbox we will set the allowFileRestoreFlag on each database.

 

[PS] C:>Set-MailboxDatabase -Identity NEW-DB1 -AllowFileRestore:$TRUE

 

Once the allowFileRestoreFlag has been set, change the database and log file paths for the new database object to match the migrated storage.  It is very important that when setting the EDB file path that you use the correct file name.  The paths do not have to be and may not be the same as they were on the original server depending on the configuration used in Step 8.

 

Use Move-DatabasePath to set the database and log file paths, as shown below.

 

[PS] C:>Move-DatabasePath NEW-DB1 -LogFolderPath f:DB1 -EdbFilePath g:DB1DB1.edb -ConfigurationOnly:$TRUE -Confirm:$FALSE

Confirm
This operation will skip the safety check and make the change to Active Directory directly. Do you want to continue?

 

Be sure to allow ample time for Active Directory replication to occur. Then, mount the database using Mount-Database.

 

[PS] C:>Mount-Database NEW-DB1

 

If the command completes successfully, the database mount status can be verified with Get-MailboxDatabase –Status.

 

[PS] C:>Get-MailboxDatabase -Identity NEW-DB1 -Status | fl *mounted

MountedOnServer : MBX-2A.exchange.msft
Mounted         : True

 

Although the database is mounted, mailboxes still reference the original dismounted database in the original database availability group.

 

Step 10:  Move mailboxes to reference the migrated database.

Begin the process of restoring mailbox access by moving the mailboxes from the original database to the new database.  This is accomplished using Get-Mailbox and Set-Mailbox.

 

[PS] C:>Get-Mailbox -Database DB1 | Set-Mailbox -Database NEW-DB1

Confirm
Rehoming mailbox "exchange.msft/LoadGen Objects/Users/MBX-1B/DB1/MBX-1B 0B63EF06-LGU000001" to database "NEW-DB1". This
operation will only modify the mailbox's Active Directory configuration. Be aware that the current mailbox content
will become inaccessible to the user.
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): a

 

After allowing sufficient time for Active Directory replication, users should be able to access their mailboxes. Transport services may need to be restarted to force re-categorization of messages to deliver to new servers.

 

Step 11:  (Optional):  Migrate storage associated with other database copies.

 

This step is optional if all storage for all database copies was migrated in step 8. In this step, complete the migration of storage from the original servers to the new servers that will house the database copies. It is important that all paths on the new servers match, so pay careful attention to how the disks are presented and how driver letters / mount points are assigned.

 

Step 12:  Add database copies of new databases to additional DAG nodes using migrated storage.

After storage has been completely migrated, the original databases should now be available on servers in the new DAG. Using Add-MailboxDatabaseCopy, you can re-instate passive copies of the database using the databases that were migrated from the original DAG. The Replication service will match these databases to the new log file stream and begin log file replay. If truncation and / or replay lag was previously configured, the copies may be added with the lag at this time.

 

[PS] C:>Add-MailboxDatabaseCopy NEW-DB1 -MailboxServer MBX-2B
[PS] C:>Add-MailboxDatabaseCopy NEW-DB1 -MailboxServer MBX-2C -ReplayLagTime 7.0:0:0

 

The success of these operations can be validated with Get-MailboxDatabaseCopyStatus.

 

[PS] C:>Get-MailboxDatabaseCopyStatus NEW-DB1*

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
NEW-DB1MBX-2A                                Mounted         0         0                                  Healthy
NEW-DB1MBX-2B                                Healthy         0         0           1/29/2014 8:59:19 PM   Crawling
NEW-DB1MBX-2C                                Healthy         0         162         1/29/2014 8:59:19 PM   Crawling

 

Once this procedure has been successfully completed on all database copies, the original servers can be decommissioned, if necessary.