At any given time, in every database availability group (DAG), there is one member that is responsible for the coordination of database actions across the DAG. This member is known as the Primary Active Manager (PAM). The current PAM can be determined by using Get-DatabaseAvailabilityGroup –Status, as shown below.

 

[PS] C:>Get-DatabaseAvailabilityGroup -Status -Identity DAG | fl name,primaryActiveManager

Name                 : DAG
PrimaryActiveManager : MBX-1

 

The mailbox server that is the PAM is always the current owner of the Cluster Core Resource group.

 

[PS] C:>Get-ClusterGroup -Cluster MBX-1

Name                                    OwnerNode                               State
—-                                    ———                               —–
Available Storage                       MBX-3                                   Offline
Cluster Group                           MBX-1                                   Online

 

The cluster group may contain several cluster resources. The PAM does not depend on the state of any of the resources in this group, and the PAM role will always be assigned to the node that owns the Cluster Group.

 

The Cluster Group can be moved between members using the cluster management tools. 

 

Windows 2008 R2 / Windows 2012 / Windows 2012 R2:

Command Line:

Cluster.exe cluster <DAGNAME> group “Cluster Group” /moveto:<NODE>

 

Powershell:

Move-ClusterGroup –cluster <DAGNAME> –name “Cluster Group” –node:<NODE>

*Note:  In Windows Server 2012 and Windows Server 2012 R2, Powershell is the preferred way to manage clusters.

 

Each DAG member that does not own the cluster group is a Standby Active Manager (SAM).  When the cluster group is moved between nodes, a notification process detects that the Cluster Group owner has changed. This triggers detection logic to determine the new PAM.  In this example, the cluster group is moved from MBX-1 to MBX-2.

 

PS C:> Move-ClusterGroup -Name "Cluster Group" -Node MBX-2

Name                                    OwnerNode                               State
—-                                    ———                               —–
Cluster Group                           MBX-2                                   Online

 

A review of Microsoft-Exchange-HighAvailability/Operational crimson channel shows the promotion of MBX-2 from SAM to PAM:

 

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      227
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-2.domain.com
Description:
Active manager configuration change detected. (PreviousRole='SAM', CurrentRole='PAM', ChangeFlags='Role, CurrentPAM', LastError='<none>')

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:48 AM
Event ID:      111
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-2.domain.com
Description:
Active manager role changed from 'SAM' to 'PAM'

 

A review of the Microsoft-Exchange-HighAvailability/Operational crimson channel shows the demotion of MBX-1 from PAM to SAM:

 

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      227
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-1.domain.com
Description:
Active manager configuration change detected. (PreviousRole='PAM', CurrentRole='SAM', ChangeFlags='Role, CurrentPAM', LastError='<none>')

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      111
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-1.domain.com
Description:
Active manager role changed from 'PAM' to 'SAM'

 

Other servers within the DAG also acknowledge the move but continue to maintain their existing roles:

 

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      227
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-3.domain.com
Description:
Active manager configuration change detected. (PreviousRole='SAM', CurrentRole='SAM', ChangeFlags='CurrentPAM', LastError='<none>')

 

The Cluster service is also responsible for automatic arbitration of this group.  Automatic arbitration may occur for a number of reasons including: 

 

• The failure of a member
• The failure of a resource contained within the Cluster group

 

In most cases, Exchange administrators should not be concerned with the owner of the cluster group or the node designated as the PAM.  This is true even for DAGs that span multiple sites where the PAM may be a node in a distant datacenter.

 

In recent weeks I have fielded several questions from administrators that are concerned with which member owns the PAM. Questions like:

 

• Should I set a preferred owner on the cluster group so that nodes in my primary datacenter are preferred over my disaster recovery datacenter?
• How do I prevent a server in the disaster recovery datacenter from becoming the PAM?
• Can I pause my nodes in cluster to prevent them from becoming the PAM?
• Should I remove possible owner on the cluster name to prevent it from coming online on a server in my disaster recovery datacenter?

 

All of these questions require modifying properties of the cluster core resource group.  By default Exchange establishes the desired settings on the cluster core resource group.  Modifying these settings is typically not necessary and can sometimes cause undesired results.  For example, recently a customer paused all of the members in a disaster recovery datacenter to prevent those servers from becoming the PAM.  This worked very well in preventing arbitration of the cluster core resource group to these nodes, until instability of the Cluster service in the primary datacenter resulted in no members being able to take ownership the cluster core resources.  In this instance the PAM was lost, and coordination of database activities across the DAG failed. 

 

The Cluster service is designed to allow the cluster group to arbitrate freely across nodes.  Attempts to modify the failover behavior, or prevent failure between nodes, can yield undesired results and potential instability.  As a result, we recommend that you leave the default out-of-box settings intact, unless you are directed by Microsoft Customer Services and Support to change them.

Although Exchange no longer uses shared storage, it can be deployed in clustered environments where shared storage is used. The storage is not actually shared between nodes as was the case with traditional clustering. Instead, the storage is presented through traditional shared storage controllers including fibre channel and iSCSI.

 

When creating a cluster with Windows Server 2012 or Windows Server 2012 R2, storage found on a shared bus is not automatically added to the cluster. When adding a node to an existing cluster the administrator is presented with an option to add shared storage automatically.  

 

 

PS C:> Add-ClusterNode -Cluster TEST -Name MBX-2

 

Allowing this option to remain checked, which is the default, will result in the Cluster service automatically adding all storage found on a shared storage bus to the cluster and clustered storage (even for disks that are not shared between nodes).  The same behavior is observed when using the Add-ClusterNode cmdlet. 

 

The disks can be observed within Failover Cluster Manager:

 

 

Physical disk resources can also be reviewed using Get-ClusterResource:

 

PS C:UsersAdministrator.EXCHANGE> Get-ClusterResource

Name                          State                         OwnerGroup                    ResourceType
—-                          —–                         ———-                    ————
Cluster Disk 1                Online                        Cluster Group                 Physical Disk
Cluster Disk 2                Online                        Available Storage             Physical Disk
Cluster Disk 3                Offline                       Available Storage             Physical Disk
Cluster Disk 4                Offline                       Available Storage             Physical Disk
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

 

Administrators can prevent the addition of the shared disks by unchecking “Add all eligible storage to the cluster” or by using the –NoStorage option with Add-ClusterNode.

 

 

PS C:> Add-ClusterNode -Cluster TEST -Name MBX-2 -NoStorage
Report file location: C:WindowsclusterReportsAdd Node Wizard 6252c9cd-5117-474b-bb7f-d117a98759ee on 2014.07.20 At 05.31.26.mht

 

This configuration can be validated using Get-ClusterResource.

 

PS C:UsersAdministrator.EXCHANGE> Get-ClusterResource

Name                          State                         OwnerGroup                    ResourceType
—-                          —–                         ———-                    ————
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

With Windows Server 2008 R2, storage found on a shared bus is not automatically added to a cluster during creation or when a node is added. The behavior is the same whether you are creating the cluster or adding the node with Failover Cluster Manager or with PowerShell. The confirmation dialog has no “Add all eligible storage to the cluster” option.

 

 

Additionally the Add-ClusterNode cmdlet does not have a –NoStorage option. 

 

PS C:> Add-ClusterNode -Cluster Cluster -Name Node-2 -noStorage
Add-ClusterNode : A parameter cannot be found that matches parameter name 'noStorage'.
At line:1 char:57
+ Add-ClusterNode -Cluster Cluster -Name Node-2 -noStorage <<<<
    + CategoryInfo          : InvalidArgument: (:) [Add-ClusterNode], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.FailoverClusters.PowerShell.AddClusterNodeCommand

 

As the membership is modified within the cluster, the lack of clustered disks can be validated with Get-ClusterResource, as well as with Failover Cluster Manager.

 

PS C:> Get-ClusterResource -Cluster Cluster

Name                          State                         Group                         ResourceType
—-                          —–                         —–                         ————
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

 

 

At this point, you’re probably wondering why I am writing a blog post about shared storage and Exchange, since database availability groups (DAGs) don’t use shared storage.

Over the course of the last few weeks, I have reviewed some DAG configurations where physical disk resources exist within the cluster. This is not a desired configuration. When disks are added to the cluster, it is the responsibility of the cluster disk driver to manage access to these resources. In these cases, checking Disk Management shows that the disks have a status of reserved. This status indicates that the storage is no longer under the control of Windows, but is instead being managed by the cluster disk driver.

 

 

Overall this causes several issues. For example, if the Cluster service fails for any reason, this makes storage in accessible to Exchange. The drive letters and mount point mappings are the same across each node even though they do not match the same physical disk. This causes confusion within the cluster and it can lead to storage instability.

 

Correcting this condition is as simple as removing the physical disk resources from cluster. This can be done using either Failover Cluster Manager or PowerShell. I recommend performing this operation during a maintenance period as it can result in the storage being temporarily inaccessible while it transitions from cluster control to Windows partition manager control.

 

At this point, you’re probably wondering how a DAG’s cluster can end up with storage shared that is controlled by the cluster.

There are actually a couple of causes. In some cases, it happens because the Exchange cmdlets failed (for example, Add-DatabaseAvailabilityGroupServer fails to successfully add a DAG member). In other cases, it happens because the cluster is rebuilt as part of a site activation process. When using Failover Cluster Manager to perform this operation you must ensure that the “Add all eligible storage to the cluster” option is unchecked in Failover Cluster Manager or that you use the –NoStorage option in PowerShell.

 

In the cases I have been involved with, it was determined that cluster membership was adjusted using Failover Cluster Manager without unchecking the add storage option. It is important for administrators to be aware of this new default option and ensure that if this condition is encountered, it is corrected as soon as possible.

Exchange Server and Office 365 offer many different options for forwarding messages to different recipients. Some of these options exist for users and others are for administrators. Administrators can also control how forwarding is handled within the organization.

 

The forwarding options available to clients and administrators are described below. Each of these methods has both pros and cons when implemented:

 

• Users can create a forwarding rule within Outlook or Outlook Web App
• An administrator can create a client rule for forwarding
• An administrator can set the ForwardingSMTPAddress parameter on a mailbox
• An administrator can set the ForwardingAddress parameter on a mailbox

 

In this post, I’m going to review each of these methods, the pros and cons, and the administrator controls in place for these options.

 

Forwarding rule within Outlook and Outlook Web App:

Users can create Inbox rules in both Outlook and Outlook Web App (OWA). One of the options is a rule to forward messages to a different recipient, which can be an internal or external recipient. In the example below, a rule has been created that forwards all messages received in the mailbox to an external recipient.

 

 

Client-side rules allow a great deal of flexibility when establishing forwarding. For example, say you want to forward emails to another internal or external recipient based on who sent the messages. The rules wizard allows this level of granularity. From an administrative standpoint client-side rules can also be helpful since they can be managed by the user. This obviates the need for administrators to be involved in the day-to-day management of these rules.

 

Client-side rules though are not immediately obvious to administrators. This may cause issues in environments that want to control where their data is stored and how it is transmitted. No specific permissions are required for users to create rules and there exists no method to limit this type of rule. Messages received by the destination mailbox look like forwarded messages; they have the same presentation as if the end-user selected the forward option on the message and specified the recipient.

 

 

 

Message tracking also confirms that the sender is the mailbox with the forwarding rule enabled and the recipient is the address specified on the forwarding rule.

 

 

 

This may cause issues in message handling, as subsequent replies would go to the mailbox the message was forwarded from rather than the original sender of the message.

 

When a message is forwarded by a client-side rule an additional header is added to the message: X-MS-Exchange-Organization-AutoForwarded. When the value of this header is set to TRUE this signifies the message was created by auto-forwarding.

 

Forwarding rule created by the administrator:

In Exchange 2010 and Exchange 2013, administrators have the New-InboxRule cmdlet, which allows them to create client-side rules within a mailbox. To create a forwarding rule that matches the previous rules wizard example you would run a command similar to the following:

 

PS C:> New-InboxRule -Name ForwardSharon -Mailbox 2148 -ForwardTo:SHARON@domain.com

Name                          Enabled                       Priority                      RuleIdentity
—-                          ——-                       ——–                      ————
ForwardSharon                 True                          1                             11219471045587107842

 

When using the Outlook client and viewing the rules in this mailbox, the administrator-created rule is present.

 

 

This rule has the same considerations and outcomes as a rule created by a user in Outlook; the only difference is that an administrator added it to the mailbox.

 

Forwarding using the forwardingSMTPAddress parameter of a mailbox:

Each mailbox has a parameter named forwardingSMTPAddress. This parameter is set by the administrator using Set-Mailbox. Any SMTP address can be specified in this parameter.

 

PS C:> Set-Mailbox 2148 -ForwardingSmtpAddress sharon@domain.com

PS C:> Get-Mailbox 2148 | fl name,forwardingSMTPAddress,delivertomailboxandforward

Name                       : Tim McMichael
ForwardingSmtpAddress      : smtp:sharon@domain.com
DeliverToMailboxAndForward : False

When this type of forwarding is used, the user has no control and the forwarding process must be administratively managed. In addition, there is no granularity with this implementation; all messages for the mailbox are forwarded to the specified SMTP address. Because the field allows any SMTP address to be specified there is no need to create a mail-enabled object within your directory to establish this type of forwarding.

 

Messaging handling is also different with this type of forwarding. When the message arrives in the destination mailbox it appears as if it was sent directly to this mailbox.  The TO field of the message reflects the original recipient, and the FROM field reflects the original sender. The message does not appear as forwarded. 

 

 

Message tracking shows that the sender of the message is the original sender and the recipient is the forwarding address.

 

 

This allows the recipient to reply to the message and have the message sent directly to the original sender. It is important to understand this behavior in scenarios where compliance or journaling is used, as subsequent replies will bypass the original organization thereby bypassing journaling and compliance processes.

 

The message header also has appended the X-MS-Exchange-Organization-AutoForwarded header. When the value of this header is set to TRUE, in indicates that the message was created by auto-forwarding.

 

Forwarding using the forwardingAddress parameter of a mailbox:

The forwardingAddress parameter of a mailbox is also managed by an administrator using Set-Mailbox. Unlike forwardingSMTPAddress, the forwarding address parameter must be an object that is defined within the directory. Attempting to set this value to an object not within the directory returns an error.

 

PS C:> Set-Mailbox 2148 -ForwardingAddress sharon@domain.com
Couldn’t find object “sharon@domain.com”. Please make sure that it was spelled correctly or specify a different
object.
    + CategoryInfo          : NotSpecified: (:) [Set-Mailbox], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=BN1PR06MB101,RequestId=3374349c-1924-43b2-bb21-d2243774a1d4,TimeStamp=7/20/2014
   5:37:29 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] F2A1EE0E,Microsoft.Exchange.Management.Reci
  pientTasks.SetMailbox
    + PSComputerName        : pod51043psh.outlook.com

 

The forwarding address parameter can be another mailbox-enabled object, a mail-enabled user, or a mail contact.  In this example, I have created a mail contact for my external recipient. 

 

PS C:> New-MailContact -Name Sharon -ExternalEmailAddress sharon@domain.com

Name                      Alias                                          RecipientType
—-                      —–                                          ————-
Sharon                    Sharon                                         MailContact

 

I then enabled the forwardingAddress parameter on the mail contact.

 

PS C:> Set-Mailbox 2148 -ForwardingAddress Sharon

 

The setting was validated with Get-Mailbox.

 

PS C:> Get-Mailbox 2148 | fl name,forwardingAddress,delivertomailboxandforward

Name                       : Tim McMichael
ForwardingAddress          : Sharon
DeliverToMailboxAndForward : False

 

Similar to the fowardingSMTPAdderss parameter the forwardingAddress parameter must be managed by the administrator. The end-user has no control over this specific parameter and the end-user will not know that forwarding is enabled. Unlike client rules, the use of the forwardingAddress parameter applies to all messages received to the destination mailbox; there is no way to allow this function to work on specific messages.

 

Messaging handling is also similar to using the fowardingSMTPAddress parameter. The message arrives in the destination mailbox as if it was addressed directly to that mailbox. The TO address shows the original recipient and the FROM address shows the original sender. 

 

 

This allows the forwarded recipient to reply to the message and have it returned directly to the original sender. Messaging tracking also shows that the sender is the original sender and the recipient is the address configured for forwarding.

 

 

It is important to understand this behavior in scenarios where compliance or journaling is used as subsequent replies will bypass the original organization thereby bypassing journaling and compliance processes.

 

In this scenario the X-MS-Exchange-Organization-AutoForwarded header is not stamped on the message.

 

Autoforwarding and delivertoandforward:

Each mailbox also has a parameter called deliverToAndForward. This allows the administrator to specify if the message should also be delivered to the mailbox where forwarding is enabled. This setting applies ONLY to forwarding that is configured by using the forwardingAddress or forwardingSMTPAddress parameters. Client-side rules are unaware of this setting.

 

Autoforwarding and remoteDomains:

Within Exchange and Office 365, administrators can create remote domains. By default, every tenant leverages the ‘*’ domain. When a specific remote domain does not exist, the ‘*’ remote domain setting are applied to the message.

 

A property of a remote domain is autoForwardEnabled property. This allows administrators to define if auto-forwarding is allowed on messages destined to the domain specified.

 

PS C:> Get-RemoteDomain  | fl name,domainname,autoForwardEnabled

Name               : Default
DomainName         : *
AutoForwardEnabled : True

 

By default auto-forwarding is allowed to all domains. Administrators can use the remote domain settings to control how forwarding outside their organization is handled. Please note: this does not change how forwarding is handled within the organization.

 

In this example, I am creating a specific remote domain rule and have disabled auto-forwarding. 

 

PS C:> New-RemoteDomain -Name ExternalDomain -DomainName domain.com

Name                           DomainName                                   AllowedOOFType
—-                           ———-                                   ————–
ExternalDomain                 domain.com                                   External

 

PS C:> Set-RemoteDomain -Identity ExternalDomain -AutoForwardEnabled:$FALSE

 

The settings can be verified with get-remoteDomain:

 

PS C:> Get-RemoteDomain ExternalDomain | fl domainname,autoforwardenabled

DomainName         : domain.com
AutoForwardEnabled : False

 

To test this feature I set the forwardingAddress on the mailbox to a recipient in the domain.com domain. When a message is addressed to the mailbox where forwarding is enabled, it never makes it to the destination. The same behavior would occur if a client-side forwarding rule is used.

 

When a client-side rule or the forwardingSMTPAddress is used, the mail flow process stamps the X-MS-Exchange-Organization-AutoForwarded to TRUE. This header is not viewable on a message as the transport header firewall removes it. When the message is processed by the Transport service, if the remote domain specifically blocks auto-forwarding and the auto-forward header is present and set to TRUE, the message is turfed, and no NDR is generated. If no form of journaling is present and the deliverToAndForward setting of the mailbox is set to FALSE, the message is effectively lost.

 

When looking at message tracking we can see that the message failed to reach its destination. The message delivery status is NONE and no send events are noted.

 

 

 

Using the same remote domain, the mailbox is configured with a forwardingAddress instead. The mail contact used as the forwardingAddress object has an external email address in domain.com. When sending a message to the mailbox where forwarding is enabled, the mail is successfully delivered. This is also confirmed by message tracking.

 

 

In this example, the transport process does not use the X-MS-Exchange-Organization-AutoForwarded but instead adds the forwarded address as a recipient during a different stage of message processing. As the message traverses the transport stack it is not detected as a forwarded message and is therefore allowed to arrive at desired recipient.

 

Utilizing the forwardingAddress scenario is an excellent way for an administrator to bypass forwarding settings on remote domains while controlling when message forwarding is allowed.

 

We routinely receive requests on how to enable granular message forwarding or only allow specific users to use auto-forwarding capabilities. Unfortunately, with the forwarding settings being tied to remote domains it is very difficult to enable forwarding for only a subset of users or where granular forwarding on messages attributes is desired. To allow clients to create auto-forwarding rules would mean the remote domain would have to allow auto-forwarding for everyone. Administrators can attempt to discover and remove auto-forwarding rules using Get-InboxRule. 

 

In my experience, in most scenarios forwarding is disallowed on remote domains and enabling of forwarding is restricted to administrator configuration using the forwardingAddress parameter.

The Start-DatabaseAvailabilityGroup cmdlet is used to restore failed nodes in a Database Availability Group (DAG) by joining the evicted nodes back to the existing cluster. When a node joins a cluster it is assigned a unique decimal value ID.  For example, the first node is assigned Node ID 1, the second node Node ID 2, and so on up to 16 supported nodes in a DAG.  You can view the node ID assigned to a particular node via the registry.  Navigate to HKLM Cluster Nodes:

 

 

The subkeys are the decimal values assigned to each node.  Selecting a subkey will expose the NetBIOS node name assigned to the NodeID.  In this example Node ID 1 is assigned to server MBX-1.

 

 

The servers list of the DAG is a multi-valued attribute that represents the members of the DAG. 

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers

Name    : DAG
Servers : {MBX-4, MBX-3, MBX-2, MBX-1}

 

When Stop-DatabaseAvailabilityGroup is run, the task parses the servers list and determines if an action should be taken on that server. For example, if the next server on the list falls within the AD site being stopped, the cmdlet will take action on that server. 

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers,startedmailboxservers,stoppedmailboxservers

Name                  : DAG
Servers               : {MBX-4, MBX-3, MBX-2, MBX-1}
StartedMailboxServers : {MBX-3.domain.com, MBX-4.domain.com}
StoppedMailboxServers : {MBX-2.domain.com, MBX-1.domain.com}

 

After running Restore-DatabaseAvailabilityGroup, the nodes on the stopped servers list are evicted from the cluster.  This results in the Node IDs within the cluster registry associated with those nodes being freed. 

 

 

At this stage, the secondary site is now functional, independent of the original.  Hopefully there will be a time where the primary site is accessible and its nodes are ready to be added back to the cluster as functional members.  Start-DatabaseAvailabilityGroup is used to bring the nodes back to the cluster.

 

Occasionally when running Start-DatabaseAvailabilityGroup the following error is thrown:

 

WARNING: Server 'DAG-4' failed to be started as a member of database availability group 'DAG'. Error: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API’ "AddClusterNode() (MaxPercentage=37) failed with 0x13af. Error: The cluster node is not valid"' failed. [Server: DAG-1.domain.com]

 

D:UtilitiesERR>err 0x13af
# for hex 0x13af / decimal 5039
  ERROR_CLUSTER_INVALID_NODE                                     winerror.h
# The cluster node is not valid.
  SQL_5039_severity_16                                           sql_err
# MODIFY FILE failed. Specified size is less than current
# size.
# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x13af
# for hex 0x13af / decimal 5039
  ERROR_CLUSTER_INVALID_NODE                                     winerror.h
# The cluster node is not valid.
# 3 matches found for "0x13af"

 

Why does this error occur?

 

Most customers will never see this error, and those that do typically see this error only in testing, where the commands associated with the Datacenter Switchover process are executed in a quick sequential fashion.  When Start-DatabaseAvailabilityGroup is run the command steps through the stopped mailbox servers list.  If you look at the list carefully you will notice that the nodes most likely appear in reverse order that then they were originally added to the DAG.

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers,startedmailboxservers,stoppedmailboxservers

Name                  : DAG
Servers               : {MBX-4, MBX-3, MBX-2, MBX-1}
StartedMailboxServers : {MBX-3.domain.com, MBX-4.domain.com}
StoppedMailboxServers : {MBX-2.domain.com, MBX-1.domain.com}

In this example, MBX-1 was added before MBX-2.  This would indicate that MBX-1 was assigned Node ID 1 and MBX-2 was assigned Node ID 2.  As nodes are added to a cluster, the first available node ID is recycled. In this example, Start-DatabaseAvailabilityGroup first detects the server MBX-2, therefore this server is the first server attempted to add to the existing cluster.  The cluster service then attempts to assign Node ID 1 to MBX-2.  For a short period of time the cluster service caches the previous Node ID to Server mapping that originally existed.  Therefore, in cache, Node ID 1 is still assigned to server MBX-1.  In this instance though we are trying to add server MBX-2 and the cluster by default attempts to assign Node ID 1 – resulting in a collision between the desired Node ID assignment and what exists in cache.  This ultimately results in the invalid node ID error returned to the cmdlet.

 

In the case of using the ADsite parameter with Start-DatabaseAvailabilityGroup, adding MBX-2 fails while adding server MBX-1 succeeds.  Based on the previous explanation this is expected, the cmdlet failed to add MBX-2 and now moved to MBX-1.  MBX-1 during the join process is assigned the first available node ID, in this case Node ID 1, which matches what previously existed in cache.  Since this match existed, the join was successful.  Executing the command multiple times with the –ADSite parameter will eventually result in all nodes being successfully added to the DAG.  If you run Start-DatabaseAvailabilityGroup with the –mailboxServer parameter, the error will continue to be seen in the short term unless the server specified matches the first available node ID.

 

Again, most administrators will not encounter this error during production site activations.  The time between when Restore-DatabaseAvailablityGroup is run and the time that the primary site is available to be added back to the DAG is sufficient for the cache to expire.  In some circumstances it may become necessary to restart the Cluster service on the surviving nodes to force the cache to expire and correct this error.  You can also use Start-DatabaseAvailabilityGroup with the –mailboxServer parameter to add the nodes back in the original order they were installed.

Outlook Web App offers the ability for users to add hyper links to certain objects within a message.  Commonly this feature is utilize when converting a text statement to a particular URL.  To invoke this feature the message author would highlight a section of text and select the Insert HyperLink   action on the toolbar.

 

 

This option may not be visible depending on the organization of the reading pane in OWA – you may need to press the menu expansion button   to expose the Insert HyperLink button. 

 

Message authors may desire to insert a link to an embedded image within a message.  In this example an image was inserted in the message. 

 

 

The embedded image was selected to allow a hyperlink to be inserted.  In this instance the insert hyperlink button is unavailable.

 

 

The insert hyperlink button is not available for embedded images.  Does that mean that an embedded image cannot have a hyperlink?  No – using keyboard shortcuts message authors can add a hyperlink to an embedded message.  Selecting the picture the message author can utilizing the shortcut CRTL + L.  Using this keyboard shortcut brings up the add hyperlink dialog.

 

 

A standard URL can be entered into this dialog and OK pressed to commit the hyperlink to the image.  When the message is sent to the recipient the hyperlinked image is included.  Clicking the image will allow the message recipient to go to the destination website specified.

 

In a future revision of Office 365 the ability to utilize the add hyperlink button will be expanded to embedded images with Outlook Web App.

Customers may have users with a security principal within their domain that do not have a mailbox hosted within their messaging solution.  In this event customers choose to mail enable the user accounts.  In an Exchange environment mail enabled users accounts appear in the address list as if a local mailbox exists but mail destined for these objects are sent to the external email address stamped on the object.

 

When using directory synchronization with Office 365 these objects are subsequently replicated into the MSOL directory as user accounts.  Exchange online then detects the presence of these mail enabled user accounts and subsequently creates a mail enabled user object in the Exchange Online directory.  This is what allows the user to appear in the Exchange Online address list.

 

Let’s take a look at an example.

 

In the on premises active directory a user account is provisioned.  This account is what allows for logon privileges to the domain.

 

[PS] C:>Get-User BlogTest

Name                                                        RecipientType
—-                                                        ————-
Blog Test                                                   User

 

The account is then mail enabled using the enable-mailUser commandlet.

 

[PS] C:>Enable-MailUser BlogTest -ExternalEmailAddress user@microsoft.com

Name                                     RecipientType
—-                                     ————-
Blog Test                                MailUser

 

Using the get-mailUser commandlet we can validate the settings of this user including the external email address set on the object.

 

[PS] C:>Get-MailUser BlogTest | fl externalemailaddress,userprincipalname,displayname,emailaddresses,primarysmtpaddress
,recipienttype

ExternalEmailAddress : SMTP:user@microsoft.com
UserPrincipalName    : BlogTest@domain.com
DisplayName          : Blog Test
EmailAddresses       : {smtp:BlogTest@domain1.domain2.com, smtp:BlogTest@serviceDomain.mail.onmicrosoft.com,
                       smtp:BlogTest@domain.com, SMTP:user@microsoft.com}
PrimarySmtpAddress   : user@microsoft.com
RecipientType        : MailUser

 

When directory synchronization occurs the object can be validated in the MSOL directory using get-MSOLUser.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@Domain2 | fl DisplayName,ProxyAddresses

DisplayName    : Blog Test
ProxyAddresses : {smtp:BlogTest@ServiceDomain.onmicrosoft.com, smtp:BlogTest@Domain2,
                 smtp:BlogTest@DOMAIN1.Domain2, smtp:BlogTest@ServiceDomain.mail.onmicrosoft.com…}

 

When the provisioning process has had time to detect and operate on the object a mail user object is created in Exchange Online.  This can be validated with the get-MailUser command when connected to Exchange Online.

 

PS C:> Get-MailUser BlogTest | fl externalemailaddress,userprincipalname,displayname,emailaddresses,primarysmtpaddress,
recipientType

ExternalEmailAddress : SMTP:user@microsoft.com
UserPrincipalName    : BlogTest@domain.com
DisplayName          : Blog Test
EmailAddresses       : {SMTP:user@microsoft.com, smtp:BlogTest@servicedomain.mail.onmicrosoft.com,
                       smtp:BlogTest@domain1.domain2.com, smtp:BlogTest@domain.com…}
PrimarySmtpAddress   : user@microsoft.com
RecipientType        : MailUser

 

At this time the object is fully provisioned and will appear in the Exchange Online global address list.  Users who select this object from the global address list will successfully send email to the remote mailbox.

 

 

 

There may exist at some time a reason to remove the on premises Active Directory user account.  When this object is removed and directory synchronization is performed this should result in the removal of the MSOL user account as well as the mail user account within Exchange Online.  Let’s look at an example.

 

Using Active Directory Users and Computers the on premises AD object is deleted.  This can be validated with Get-User.

 

[PS] C:>Get-User BlogTest

The operation couldn't be performed because object 'BlogTest' couldn't be found on 'Server1.domain.domain.com'.
    + CategoryInfo          : NotSpecified: (:) [Get-User], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=MAIL,RequestId=95b95c5e-03c2-406a-9400-c490c0d7cddb,TimeStamp=5/13/2014 1:34:34PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 4549B0D2,Microsoft.Exchange.Management.RecipientTasks.GetUser
    + PSComputerName        : mail.domain.domain.com

 

This also results in the removal of the on premises mail user object.  This can be validated with Get-MailUser.

 

[PS] C:>Get-MailUser BlogTest

The operation couldn't be performed because object 'BlogTest' couldn't be found on 'Server1.domain1.domain2.com'.
    + CategoryInfo          : NotSpecified: (:) [Get-MailUser], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=MAIL,RequestId=0106f632-ad95-4f07-b054-4e749e437cfc,TimeStamp=5/13/2014 1:35:55PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 84EC86F5,Microsoft.Exchange.Management.RecipientTasks.GetMailUser
    + PSComputerName        : mail.domain1.domain2.com

 

When directory synchronization has completed we can validate the object no longer exists in the MSOL directory using get-MSOLUser.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@domain2.com | fl DisplayName,ProxyAddresses

Get-MsolUser : User Not Found.  User: BlogTest@domain2.com.
At line:1 char:1
+ Get-MsolUser -UserPrincipalName BlogTest@domain2.com | fl DisplayName,Pro …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [Get-MsolUser], MicrosoftOnlineException
    + FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UserNotFoundException,Microsoft.Online.Administration.Automation.GetUser

 

When the provisioning process has detected the deletion the mail user object is removed from Exchange Online.  This can be validated with get-MailUser.

 

PS C:> Get-MailUser BlogTest

The operation couldn't be performed because object 'BlogTest' couldn't be found on 'CO1PR06A002DC02.NAMPR06A002.prod.outlook.com'.
    + CategoryInfo          : NotSpecified: (:) [Get-MailUser], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=BN1PR06MB101,RequestId=c3cde9d7-e638-4808-8891-65d539689698,TimeStamp=5/13/2014 1:40:19 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] 782EAA9B,Microsoft.Exchange.Management.RecipientTasks.GetMailUser
    + PSComputerName        : pod51043psh.outlook.com

 

The object after deletion from the Exchange Online directory will no longer appear in the address list. 

 

 

End users who desire to email this object would have to address the message manually to the external email address.  For example:

 

 

This is where the issue arises.  In this instance the user has successfully addressed an email to the external email address but the mail is returned with a non-delivery report.

 

 

Delivery has failed to these recipients or groups:

Blog Test

The email address you entered couldn't be found. Check the recipient's email address then try to resend the message. For more tips to resolve this issue see DSN code 5.1.1 in Exchange Online. If the problem continues contact your help desk.

 

The diagnostic information states the following:

 

Diagnostic information for administrators:
Generating server: DM2PR0601MB0953.namprd06.prod.outlook.com
IMCEAEX-_o=ExchangeLabs_ou=Exchange+20Administrative+20Group+20+28FYDIBOHF23SPDLT+29_cn=Recipients_cn=0729272369574e7d945aeeecf1afd94c-Blog+20Test@namprd06.prod.outlook.com
Remote Server returned '550 5.1.1 RESOLVER.ADR.ExRecipNotFound; not found'
Original message headers:

Received: from DM2PR0601MB0953.namprd06.prod.outlook.com (25.160.25.145) by
DM2PR0601MB0953.namprd06.prod.outlook.com (25.160.25.145) with Microsoft SMTP
Server (TLS) id 15.0.944.11; Tue, 13 May 2014 13:43:53 +0000
Received: from DM2PR0601MB0953.namprd06.prod.outlook.com ([25.160.25.145]) by
DM2PR0601MB0953.namprd06.prod.outlook.com ([25.160.25.145]) with mapi id
15.00.0944.000; Tue, 13 May 2014 13:43:53 +0000
Content-Type: application/ms-tnef; name="winmail.dat"
Content-Transfer-Encoding: binary
From: Dev User <DevUser@servicedomain.onmicrosoft.com>
To: Blog Test <user@microsoft.com>
Subject: Test Message
Thread-Topic: Test Message
Thread-Index: AQHPbrFT/8JoaY496EePsUIz8dp/Tw==
Date: Tue, 13 May 2014 13:43:52 +0000
Message-ID: <1399988632393.9363@FortMillRescueSquad.onmicrosoft.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator: <1399988632393.9363@servicedomain.onmicrosoft.com>
MIME-Version: 1.0
X-Originating-IP: [167.220.151.116]
Return-Path: DevUser@serviceDomain.onmicrosoft.com

 

For many administrators this NDR is recognizable.  The same NDR is generated when an object is deleted from the environment but the email is addressed using nickname or recipient cache.  In this instance though the email was not addressed using nickname or recipient cache but rather the SMTP address was fully typed in the TO: line.  (Note:  The email address was manually removed from the nickname cache prior to addressing the email to ensure that automatic resolution does not occur.)

 

Why is the mail NDRing.  When a user account is removed from the MSOL directory the account is placed into a soft deleted state.  This can be validated with get-MSOLUser –returnedDeletedUsers.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@domain.com -ReturnDeletedUsers

UserPrincipalName                       DisplayName                             isLicensed
—————–                       ———–                             ———-
BlogTest@domain.com                     Blog Test                               False

 

While the user remains in a soft deleted state within the MSOL directory the corresponding mail user object remains in a soft deleted state within Exchange Online.  It is believed that the recipient resolvers within transport detect the presences of the soft deleted mail user object within the Exchange Online directory and subsequently produce a non-delivery report based on the state of this object.

 

This mail is legitimate though – how can the issue be resolved?

 

In order to remove the soft deleted mail user object from the Exchange Online directory the soft deleted user object must be removed from the MSOL directory.  This can be accomplished with remove-MSOLUser.

 

PS C:> Remove-MsolUser -UserPrincipalName BlogTest@domain.com -RemoveFromRecycleBin

Confirm
Continue with this operation?
[Y] Yes  [N] No  [S] Suspend  [?] Help (default is "Y"): y

 

The results can be validated with Get-MSOLUser.

 

PS C:> Get-MsolUser -UserPrincipalName BlogTest@domain.com -ReturnDeletedUsers
Get-MsolUser : User Not Found.  User: BlogTest@fortmillrescue.com.
At line:1 char:1
+ Get-MsolUser -UserPrincipalName BlogTest@fortmillrescue.com -ReturnDeletedUsers
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [Get-MsolUser], MicrosoftOnlineException
    + FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UserNotFoundException,Microsoft.Online.Administration.Automation.GetUser

 

The provisioning process will eventually detect the deletion of the object from the MSOL directory.  This will cause the soft deleted mail user object to be purged from the Exchange Online directory.  After this object is purged mail should deliver successfully to the external account when the full external email address is utilized.

 

 

At this time anytime a mail user object is removed from the on premises directory the corresponding soft deleted MSOL object would need to also be removed in order to ensure that mail addressed to the external object will immediately function.  In many cases the soft deleted MSOL object will expire, resulting in the removal of the soft deleted mail user object from the Exchange Online directory, prior to this issue being noticed by the end user population.

 

Our development teams are aware of this behavior and are considering potential future service modifications.

In both Exchange 2010 and Exchange 2013, customers can deploy one or more replication networks in a database availability group (DAG).  There can be many reasons for using a replication network, but in most cases they are used to provide a dedicated log shipping channel between members of the same DAG.

 

As documented on TechNet, when a replication network fails, replication should automatically failover to the DAG’s MAPI network:

· http://technet.microsoft.com/en-us/library/dd638104(v=exchg.150).aspx (Exchange 2013)

· http://technet.microsoft.com/en-us/library/dd638104(v=exchg.141).aspx (Exchange 2010)

 

“In the event of a failure affecting the Replication network, if the MAPI network is unaffected by the failure, log shipping and seeding operations will revert to use the MAPI network, even if the MAPI network has it's ReplicationEnabled property set to False.”

 

Log shipping operations occur by connecting to the Microsoft Exchange Replication service on the server that hosts the active database copy, on TCP port 64327 using a random ephemeral port on the passive server.  Log files are then pushed from the active server to the passive server via this channel.  There can exist several issues that result in this log shipping channel being interrupted.  For example, a firewall may block port 64327, static routes may be missing on multi-subnet replication networks, or an intermediary network device may not route traffic correctly. 

 

In the following example, we have a 4-member DAG running Exchange 2013.  There is a single active database that is replicated to three other servers.

 

 

The DAG has two networks that each has two subnets.  These networks represent the MAPI and Replication networks for the DAG.  Automatic network detection in Exchange 2013 was disabled for this example.

 

RunspaceId         : 60e6ae0f-e69d-4ae6-9fcb-8c99ea9fd21f
Name               : MapiDagNetwork
Description        :
Subnets            : {{192.168.0.0/24,Up}, {192.168.1.0/24,Up}}
Interfaces         : {{MBX-1,Up,192.168.0.11}, {MBX-2,Up,192.168.0.12}, {MBX-3,Up,192.168.1.11},
                     {MBX-4,Up,192.168.1.12}}
MapiAccessEnabled  : True
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAGMapiDagNetwork
IsValid            : True
ObjectState        : New

RunspaceId         : 60e6ae0f-e69d-4ae6-9fcb-8c99ea9fd21f
Name               : ReplicationDagNetwork01
Description        :
Subnets            : {{10.0.1.0/24,Up}, {10.0.0.0/24,Up}}
Interfaces         : {{MBX-1,Up,10.0.0.1}, {MBX-2,Up,10.0.0.2}, {MBX-3,Up,10.0.1.1}, {MBX-4,Up,10.0.1.2}}
MapiAccessEnabled  : False
ReplicationEnabled : True
IgnoreNetwork      : False
Identity           : DAGReplicationDagNetwork01
IsValid            : True
ObjectState        : New

 

Using Get-MailboxDatabaseCopyStatus with the –ConnectionStatus switch, we can verify that the Replication network is currently in use:

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -connectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

 

The router servicing the network link between the two subnets of the Replication network is shutdown.  This will block replication from succeeding over the Replication network.  What happens to the database copies?

 

[PS] C:>Get-MailboxDatabaseCopyStatus * | fl name,status

Name   : DAG-DB0MBX-1
Status : Mounted

Name   : DAG-DB0MBX-2
Status : Healthy

Name   : DAG-DB0MBX-3
Status : DisconnectedAndHealthy

Name   : DAG-DB0MBX-4
Status : DisconnectedAndHealthy

 

In this example, the database copies enter a DisconnectedAndHealthy state.  Reviewing the connection status, we note that the connection has timed out between the servers; this is expected since the route is down.

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {{MBX-2,ReplicationDagNetwork01}}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

 

Why didn’t the Replication network failover to the MAPI network in this case?  For one thing, the ability to establish a log shipping channel is not one of the criteria that the Replication service uses to determine the health of a given network.  The Replication service relies on feedback from the Cluster service regarding individual network interface status in order to determine the health of a log shipping channel.

 

For each subnet that exists on a DAG member, an associated cluster network is created.  In this example, there are 4 subnets and therefore there are 4 cluster networks.

 

[PS] C:>Get-ClusterNetwork | fl

Name  : Cluster Network 1
State : Up

Name  : Cluster Network 2
State : Up

Name  : Cluster Network 3
State : Up

Name  : Cluster Network 4
State : Up

 

 

The interface associated with each of these subnets is included in the appropriate cluster network.  Each of these interfaces has a status based on status reporting in Windows Failover Clustering.

 

[PS] C:>Get-ClusterNetworkInterface | fl

Name    : MBX-1 – LAN-A
Node    : MBX-1
Network : Cluster Network 1
State   : Up

Name    : MBX-2 – LAN-A
Node    : MBX-2
Network : Cluster Network 1
State   : Up

Name    : MBX-1 – REPL-A
Node    : MBX-1
Network : Cluster Network 2
State   : Up

Name    : MBX-2 – REPL-A
Node    : MBX-2
Network : Cluster Network 2
State   : Up

Name    : MBX-3 – REPL-B
Node    : MBX-3
Network : Cluster Network 3
State   : Up

Name    : MBX-4 – REPL-B
Node    : MBX-4
Network : Cluster Network 3
State   : Up

Name    : MBX-3 – LAN-B
Node    : MBX-3
Network : Cluster Network 4
State   : Up

Name    : MBX-4 – LAN-B
Node    : MBX-4
Network : Cluster Network 4
State   : Up

 

 

In this scenario, the Cluster service considers all of the interfaces as “UP.”  Since the interfaces are UP, the Replication service does not failover to the MAPI network even though replication cannot occur over the replication network. 

 

If the Cluster service reports a network as “FAILED” the behavior is different.  On a server hosting a passive database copy, the network cable was removed from the Replication network interface, causing the cluster to mark that interfaces as failed:

 

[PS] C:>Get-ClusterNetworkInterface | fl

Name    : MBX-1 – LAN-A
Node    : MBX-1
Network : Cluster Network 1
State   : Up

Name    : MBX-2 – LAN-A
Node    : MBX-2
Network : Cluster Network 1
State   : Up

Name    : MBX-1 – REPL-A
Node    : MBX-1
Network : Cluster Network 2
State   : Up

Name    : MBX-2 – REPL-A
Node    : MBX-2
Network : Cluster Network 2
State   : Up

Name    : MBX-3 – REPL-B
Node    : MBX-3
Network : Cluster Network 3
State   : Up

Name    : MBX-4 – REPL-B
Node    : MBX-4
Network : Cluster Network 3
State   : Failed

Name    : MBX-3 – LAN-B
Node    : MBX-3
Network : Cluster Network 4
State   : Up

Name    : MBX-4 – LAN-B
Node    : MBX-4
Network : Cluster Network 4
State   : Up

 

The copy status for the databases hosted on MBX-4 is healthy.

 

[PS] C:>Get-MailboxDatabaseCopyStatus *

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DAG-DB0MBX-1                                 Mounted         0         0                                  Healthy
DAG-DB0MBX-2                                 Healthy         0         0           4/29/2014 2:11:47 PM   Healthy
DAG-DB0MBX-3                                 Healthy         0         0           4/29/2014 2:11:47 PM   Healthy
DAG-DB0MBX-4                                 Healthy         0         0           4/29/2014 2:11:47 PM   Healthy

 

When reviewing the connection status for the databases on MBX-4, we see that the Replication service is using the MAPI network for log shipping.  Servers that have no issues with the Replication interface continue to use that interface.

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,MapiDagNetwork}
OutgoingConnections       :

 

When the interface was marked as failed, the Replication service successfully failed over to the MAPI network.

 

But what happens if the interface is not FAILED, but the interface configuration is invalid?   In this event, the Cluster service cannot pass cluster traffic between these two interfaces and the associated network is marked as partitioned.

 

[PS] C:>Get-ClusterNetwork | fl

Name  : Cluster Network 1
State : Up

Name  : Cluster Network 2
State : Up

Name  : Cluster Network 3
State : Partitioned

Name  : Cluster Network 4
State : Up

 

The interfaces within the partitioned network are marked as unreachable.

 

[PS] C:>Get-ClusterNetworkInterface | fl

Name    : MBX-1 – LAN-A
Node    : MBX-1
Network : Cluster Network 1
State   : Up

Name    : MBX-2 – LAN-A
Node    : MBX-2
Network : Cluster Network 1
State   : Up

Name    : MBX-1 – REPL-A
Node    : MBX-1
Network : Cluster Network 2
State   : Up

Name    : MBX-2 – REPL-A
Node    : MBX-2
Network : Cluster Network 2
State   : Up

Name    : MBX-3 – REPL-B
Node    : MBX-3
Network : Cluster Network 3
State   : Unreachable

Name    : MBX-4 – REPL-B
Node    : MBX-4
Network : Cluster Network 3
State   : Unreachable

Name    : MBX-3 – LAN-B
Node    : MBX-3
Network : Cluster Network 4
State   : Up

Name    : MBX-4 – LAN-B
Node    : MBX-4
Network : Cluster Network 4
State   : Up

 

 

What happens to the copies that are hosted on the servers with an unreachable interface status?

 

[PS] C:>Get-MailboxDatabaseCopyStatus * | fl name,status

Name   : DAG-DB0MBX-1
Status : Mounted

Name   : DAG-DB0MBX-2
Status : Healthy

Name   : DAG-DB0MBX-3
Status : DisconnectedAndHealthy

Name   : DAG-DB0MBX-4
Status : DisconnectedAndHealthy

 

An unreachable interface is not the same as a failed interface.  This resulted in the databases entering a disconnected state and replication not failing over to the MAPI network.  The connection status confirms this failure.

 

[PS] C:>Get-MailboxDatabaseCopyStatus * -ConnectionStatus | fl name,incominglogcopyingnetwork,outgoingconnections

Name                      : DAG-DB0MBX-1
IncomingLogCopyingNetwork :
OutgoingConnections       : {}

Name                      : DAG-DB0MBX-2
IncomingLogCopyingNetwork : {MBX-1,ReplicationDagNetwork01}
OutgoingConnections       :

Name                      : DAG-DB0MBX-3
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

Name                      : DAG-DB0MBX-4
IncomingLogCopyingNetwork : {MBX-1,,A timeout occurred while communicating with server 'MBX-1'. Error: "A connection
                            could not be completed within 15 seconds."}
OutgoingConnections       :

 

The network failure detection mechanism is the same in both Exchange 2010 and Exchange 2013, as well as the same in Windows 2008 R2, Windows 2012, and Windows 2012 R2 failover clusters.  In order for the Replication service to detect a failure of a Replication network, the operating system and Cluster service must report the underlying interface as failed.  If the Cluster service reports any other status for the interface, the Replication service will consider the network to be valid and replication will not failover to another network.

In Office 365 mailboxes are migrated to the service by configuring a migration batch.  When establishing the move configuration settings a “More Options” button was present.  The “More Options” button allowed administrators to configure the bad item limit and large item limit when the migration batch is created.

 

 

In current versions of the Office 365 / Exchange management tools the “More Options” button is missing.

 

 

The “More Options” button was removed as a part of the Office 365 / Exchange Management tools upgrade. 

 

If the bad item limit and large item limit must be adjusted, they must now be adjusted after the migration batch has been created.  These settings can only be adjusted on a migration batch that is not in progress. 

 

When creating new migration batches where these settings should be modified administrators should select the option “Manually start the batch later (by selecting it in the migration dashboard and then clicking Start)”.  This will stage the migration batch but not start it, allowing administrators to immediately adjust the bad item and large item limits.  The status of the migration batch will be “created”.

 

 

If the migration batch was created and has started it must be stopped.  This can be performed by selecting the migration batch and pressing the stop button (square button).  This will place the migration batch into a “stopped” state.

 

 

With the migration batch created or stopped the properties of the batch can be accessed by selecting the batch and pressing the edit button (pencil button).  On the properties of the batch, on the general tab, the bad item limit and large item limit are editable. 

 

 

After entering the appropriate values the save button can be utilized to commit the changes to the batch.  The batch can then be started or resumed by selecting the batch and pressing the resume button (play button). 

 

At this time we expect the “More Options” button to appear in the Office 365 / Exchange management tools in a future version. 

I recently worked with a customer that was utilizing directory synchronization to create distribution groups in their on-premises active directory and have them mail enabled within Office 365.  They noticed that when creating these distribution groups the display name in Office 365 was automatically changed and differed from the display name set on-premises.

 

Let’s take a look…

 

Using Active Directory Users and Computers a new distribution list is created.

 

 

Using the attribute editor the following attributes were manually set:

 

DisplayName:  TestDL

ProxyAddresses:  SMTP:TestDL@domain.com

Mail:  TestDL@Domain.com

 

After the distribution list was created directory synchronization was performed.  Monitoring directory synchronization we can validate the properties of the group synchronized into Office 365.

 

 

Using Powershell connected to Office 365 we can validate the properties of the MSOL Group.

 

PS C:Userstimmcmic> Get-MsolGroup -ObjectId df617972-c874-4d6d-ba73-993e73219514 | fl

ExtensionData    : System.Runtime.Serialization.ExtensionDataObject
CommonName       : TestDL
Description      :
DisplayName      : TestDL
EmailAddress     : TestDL@domain.com
Errors           : {}
GroupType        : DistributionList
IsSystem         : False
LastDirSyncTime  : 4/9/2014 2:56:34 PM
ManagedBy        :
ObjectId         : df617972-c874-4d6d-ba73-993e73219514
ProxyAddresses   : {smtp:TestDL@domain.onmicrosoft.com, SMTP:TestDL@domain.com}
ValidationStatus : Healthy

 

The attributes of the group within Office 365 and the on-premises Active Directory match after directory synchronization has completed.

 

When reviewing the properties of the same distribution list from Office 365 Exchange it is noted that the name has changed.  In this case a prefix was appended DL_:

 

PS C:Userstimmcmic> Get-DistributionGroup TestDL | fl NAME,DISPLAYNAME,EMAILADDRESSES

Name           : DL_TestDL
DisplayName    : DL_TestDL
EmailAddresses : {SMTP:TestDL@domain.com, smtp:TestDL@domain.onmicrosoft.com}

Where did the DL_ come from?  Exchange and Office 365 allow administrators to define a distribution group naming policy.  The distribution group naming policy is set in the organization’s configuration.  More information on distribution group naming policies can be found here:  https://aka.ms/Bry5y2

 

In this instance the tenant had a distribution group naming policy that utilized DL_<GroupName>:

 

PS C:Userstimmcmic> Get-OrganizationConfig | fl distributiongroupnamingpolicy

DistributionGroupNamingPolicy : DL_<GroupName>

 

When a distribution group naming policy is utilized after directory synchronization has successfully synchronized the group and the new group is provisioned within Exchange Office 365 the naming policy will be applied.  The MSOL object will continue to reflect the attributes as synchronized from the on-premises Active Directory but the Exchange object will reflect the attributes modified by the naming policy.

 

In our case it was desired that the naming policy would not be applied.  This required removing the naming policy from the organizational configuration:

 

PS C:Userstimmcmic> Set-OrganizationConfig -DistributionGroupNamingPolicy:$NULL

 

By removing the naming policy the Exchange Office 365 groups will now reflect the attributes as synchronized from the on-premises Active Directory.

Recently I was re-reading a blog post from 2012 about on Tuning Failover Cluster Network Thresholds that was posted by Elden Christensen, a Principal PM on the Windows Failover Cluster team. I think Elden’s post is a must-read for anyone planning a highly-available Exchange deployment.  One of the reasons I find this post an excellent read is that it addresses many things administrators need to understand when they decide to tune cluster heartbeat subnet delays and thresholds in Exchange environments.

 

The post first makes an excellent point in addressing that the changes to these thresholds alter the amount of time it takes to detect that a node is down.  Elden uses a great metaphor here:

 

“Think of it like your cell phone, when the other end goes silent how long are you willing to sit there going “Hello?… Hello?… Hello?” before you hang-up the phone and call the person back.  When the other end goes silent, you don’t know when or even if they will come back.”

As subnet thresholds are adjusted up, this increases the amount of time it takes to detect a failure. The higher the values, the longer it takes to detect a failure, and therefore the longer it takes to act on that failure. There is a balance between reacting quickly to a failure and providing resiliency to transient networking issues.

 

The other point that I think is worth understanding is the number of times these values are adjusted in the absence of an analysis or correction of underlying networking issues.  Elden sums this up, too, and I could not agree with him more:

 

“It critical to recognize that cranking up the thresholds to high values does not fix nor resolve the transient network issue, it simply masks the problem by making health monitoring less sensitive. The #1 mistake made broadly by customers is the perception of not triggering cluster health detection means the issue is resolved (which is not true!). I like to think of it, that just because you choose not to go to the doctor it does not mean you are healthy. In other words, the lack of someone telling you that you have a problem does not mean the problem went away.”

I often find myself in conversations with customers who have changed these values and have the perception that something is “fixed.” There are legitimate cases where these values need to be changed – but I always encourage a networking analysis enables you to understand what issues you are facing and how adjusting these values would help. Unfortunately, it seems that adjusting these thresholds without this understanding is far more common than it should be.

 

I strongly encourage all Exchange administrators to read Elden’s post.