Although Exchange no longer uses shared storage, it can be deployed in clustered environments where shared storage is used. The storage is not actually shared between nodes as was the case with traditional clustering. Instead, the storage is presented through traditional shared storage controllers including fibre channel and iSCSI.

 

When creating a cluster with Windows Server 2012 or Windows Server 2012 R2, storage found on a shared bus is not automatically added to the cluster. When adding a node to an existing cluster the administrator is presented with an option to add shared storage automatically.  

 

 

PS C:> Add-ClusterNode -Cluster TEST -Name MBX-2

 

Allowing this option to remain checked, which is the default, will result in the Cluster service automatically adding all storage found on a shared storage bus to the cluster and clustered storage (even for disks that are not shared between nodes).  The same behavior is observed when using the Add-ClusterNode cmdlet. 

 

The disks can be observed within Failover Cluster Manager:

 

 

Physical disk resources can also be reviewed using Get-ClusterResource:

 

PS C:UsersAdministrator.EXCHANGE> Get-ClusterResource

Name                          State                         OwnerGroup                    ResourceType
—-                          —–                         ———-                    ————
Cluster Disk 1                Online                        Cluster Group                 Physical Disk
Cluster Disk 2                Online                        Available Storage             Physical Disk
Cluster Disk 3                Offline                       Available Storage             Physical Disk
Cluster Disk 4                Offline                       Available Storage             Physical Disk
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

 

Administrators can prevent the addition of the shared disks by unchecking “Add all eligible storage to the cluster” or by using the –NoStorage option with Add-ClusterNode.

 

 

PS C:> Add-ClusterNode -Cluster TEST -Name MBX-2 -NoStorage
Report file location: C:WindowsclusterReportsAdd Node Wizard 6252c9cd-5117-474b-bb7f-d117a98759ee on 2014.07.20 At 05.31.26.mht

 

This configuration can be validated using Get-ClusterResource.

 

PS C:UsersAdministrator.EXCHANGE> Get-ClusterResource

Name                          State                         OwnerGroup                    ResourceType
—-                          —–                         ———-                    ————
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

With Windows Server 2008 R2, storage found on a shared bus is not automatically added to a cluster during creation or when a node is added. The behavior is the same whether you are creating the cluster or adding the node with Failover Cluster Manager or with PowerShell. The confirmation dialog has no “Add all eligible storage to the cluster” option.

 

 

Additionally the Add-ClusterNode cmdlet does not have a –NoStorage option. 

 

PS C:> Add-ClusterNode -Cluster Cluster -Name Node-2 -noStorage
Add-ClusterNode : A parameter cannot be found that matches parameter name 'noStorage'.
At line:1 char:57
+ Add-ClusterNode -Cluster Cluster -Name Node-2 -noStorage <<<<
    + CategoryInfo          : InvalidArgument: (:) [Add-ClusterNode], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.FailoverClusters.PowerShell.AddClusterNodeCommand

 

As the membership is modified within the cluster, the lack of clustered disks can be validated with Get-ClusterResource, as well as with Failover Cluster Manager.

 

PS C:> Get-ClusterResource -Cluster Cluster

Name                          State                         Group                         ResourceType
—-                          —–                         —–                         ————
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

 

 

At this point, you’re probably wondering why I am writing a blog post about shared storage and Exchange, since database availability groups (DAGs) don’t use shared storage.

Over the course of the last few weeks, I have reviewed some DAG configurations where physical disk resources exist within the cluster. This is not a desired configuration. When disks are added to the cluster, it is the responsibility of the cluster disk driver to manage access to these resources. In these cases, checking Disk Management shows that the disks have a status of reserved. This status indicates that the storage is no longer under the control of Windows, but is instead being managed by the cluster disk driver.

 

 

Overall this causes several issues. For example, if the Cluster service fails for any reason, this makes storage in accessible to Exchange. The drive letters and mount point mappings are the same across each node even though they do not match the same physical disk. This causes confusion within the cluster and it can lead to storage instability.

 

Correcting this condition is as simple as removing the physical disk resources from cluster. This can be done using either Failover Cluster Manager or PowerShell. I recommend performing this operation during a maintenance period as it can result in the storage being temporarily inaccessible while it transitions from cluster control to Windows partition manager control.

 

At this point, you’re probably wondering how a DAG’s cluster can end up with storage shared that is controlled by the cluster.

There are actually a couple of causes. In some cases, it happens because the Exchange cmdlets failed (for example, Add-DatabaseAvailabilityGroupServer fails to successfully add a DAG member). In other cases, it happens because the cluster is rebuilt as part of a site activation process. When using Failover Cluster Manager to perform this operation you must ensure that the “Add all eligible storage to the cluster” option is unchecked in Failover Cluster Manager or that you use the –NoStorage option in PowerShell.

 

In the cases I have been involved with, it was determined that cluster membership was adjusted using Failover Cluster Manager without unchecking the add storage option. It is important for administrators to be aware of this new default option and ensure that if this condition is encountered, it is corrected as soon as possible.

Exchange Server and Office 365 offer many different options for forwarding messages to different recipients. Some of these options exist for users and others are for administrators. Administrators can also control how forwarding is handled within the organization.

 

The forwarding options available to clients and administrators are described below. Each of these methods has both pros and cons when implemented:

 

• Users can create a forwarding rule within Outlook or Outlook Web App
• An administrator can create a client rule for forwarding
• An administrator can set the ForwardingSMTPAddress parameter on a mailbox
• An administrator can set the ForwardingAddress parameter on a mailbox

 

In this post, I’m going to review each of these methods, the pros and cons, and the administrator controls in place for these options.

 

Forwarding rule within Outlook and Outlook Web App:

Users can create Inbox rules in both Outlook and Outlook Web App (OWA). One of the options is a rule to forward messages to a different recipient, which can be an internal or external recipient. In the example below, a rule has been created that forwards all messages received in the mailbox to an external recipient.

 

 

Client-side rules allow a great deal of flexibility when establishing forwarding. For example, say you want to forward emails to another internal or external recipient based on who sent the messages. The rules wizard allows this level of granularity. From an administrative standpoint client-side rules can also be helpful since they can be managed by the user. This obviates the need for administrators to be involved in the day-to-day management of these rules.

 

Client-side rules though are not immediately obvious to administrators. This may cause issues in environments that want to control where their data is stored and how it is transmitted. No specific permissions are required for users to create rules and there exists no method to limit this type of rule. Messages received by the destination mailbox look like forwarded messages; they have the same presentation as if the end-user selected the forward option on the message and specified the recipient.

 

 

 

Message tracking also confirms that the sender is the mailbox with the forwarding rule enabled and the recipient is the address specified on the forwarding rule.

 

 

 

This may cause issues in message handling, as subsequent replies would go to the mailbox the message was forwarded from rather than the original sender of the message.

 

When a message is forwarded by a client-side rule an additional header is added to the message: X-MS-Exchange-Organization-AutoForwarded. When the value of this header is set to TRUE this signifies the message was created by auto-forwarding.

 

Forwarding rule created by the administrator:

In Exchange 2010 and Exchange 2013, administrators have the New-InboxRule cmdlet, which allows them to create client-side rules within a mailbox. To create a forwarding rule that matches the previous rules wizard example you would run a command similar to the following:

 

PS C:> New-InboxRule -Name ForwardSharon -Mailbox 2148 -ForwardTo:SHARON@domain.com

Name                          Enabled                       Priority                      RuleIdentity
—-                          ——-                       ——–                      ————
ForwardSharon                 True                          1                             11219471045587107842

 

When using the Outlook client and viewing the rules in this mailbox, the administrator-created rule is present.

 

 

This rule has the same considerations and outcomes as a rule created by a user in Outlook; the only difference is that an administrator added it to the mailbox.

 

Forwarding using the forwardingSMTPAddress parameter of a mailbox:

Each mailbox has a parameter named forwardingSMTPAddress. This parameter is set by the administrator using Set-Mailbox. Any SMTP address can be specified in this parameter.

 

PS C:> Set-Mailbox 2148 -ForwardingSmtpAddress sharon@domain.com

PS C:> Get-Mailbox 2148 | fl name,forwardingSMTPAddress,delivertomailboxandforward

Name                       : Tim McMichael
ForwardingSmtpAddress      : smtp:sharon@domain.com
DeliverToMailboxAndForward : False

When this type of forwarding is used, the user has no control and the forwarding process must be administratively managed. In addition, there is no granularity with this implementation; all messages for the mailbox are forwarded to the specified SMTP address. Because the field allows any SMTP address to be specified there is no need to create a mail-enabled object within your directory to establish this type of forwarding.

 

Messaging handling is also different with this type of forwarding. When the message arrives in the destination mailbox it appears as if it was sent directly to this mailbox.  The TO field of the message reflects the original recipient, and the FROM field reflects the original sender. The message does not appear as forwarded. 

 

 

Message tracking shows that the sender of the message is the original sender and the recipient is the forwarding address.

 

 

This allows the recipient to reply to the message and have the message sent directly to the original sender. It is important to understand this behavior in scenarios where compliance or journaling is used, as subsequent replies will bypass the original organization thereby bypassing journaling and compliance processes.

 

The message header also has appended the X-MS-Exchange-Organization-AutoForwarded header. When the value of this header is set to TRUE, in indicates that the message was created by auto-forwarding.

 

Forwarding using the forwardingAddress parameter of a mailbox:

The forwardingAddress parameter of a mailbox is also managed by an administrator using Set-Mailbox. Unlike forwardingSMTPAddress, the forwarding address parameter must be an object that is defined within the directory. Attempting to set this value to an object not within the directory returns an error.

 

PS C:> Set-Mailbox 2148 -ForwardingAddress sharon@domain.com
Couldn’t find object “sharon@domain.com”. Please make sure that it was spelled correctly or specify a different
object.
    + CategoryInfo          : NotSpecified: (:) [Set-Mailbox], ManagementObjectNotFoundException
    + FullyQualifiedErrorId : [Server=BN1PR06MB101,RequestId=3374349c-1924-43b2-bb21-d2243774a1d4,TimeStamp=7/20/2014
   5:37:29 PM] [FailureCategory=Cmdlet-ManagementObjectNotFoundException] F2A1EE0E,Microsoft.Exchange.Management.Reci
  pientTasks.SetMailbox
    + PSComputerName        : pod51043psh.outlook.com

 

The forwarding address parameter can be another mailbox-enabled object, a mail-enabled user, or a mail contact.  In this example, I have created a mail contact for my external recipient. 

 

PS C:> New-MailContact -Name Sharon -ExternalEmailAddress sharon@domain.com

Name                      Alias                                          RecipientType
—-                      —–                                          ————-
Sharon                    Sharon                                         MailContact

 

I then enabled the forwardingAddress parameter on the mail contact.

 

PS C:> Set-Mailbox 2148 -ForwardingAddress Sharon

 

The setting was validated with Get-Mailbox.

 

PS C:> Get-Mailbox 2148 | fl name,forwardingAddress,delivertomailboxandforward

Name                       : Tim McMichael
ForwardingAddress          : Sharon
DeliverToMailboxAndForward : False

 

Similar to the fowardingSMTPAdderss parameter the forwardingAddress parameter must be managed by the administrator. The end-user has no control over this specific parameter and the end-user will not know that forwarding is enabled. Unlike client rules, the use of the forwardingAddress parameter applies to all messages received to the destination mailbox; there is no way to allow this function to work on specific messages.

 

Messaging handling is also similar to using the fowardingSMTPAddress parameter. The message arrives in the destination mailbox as if it was addressed directly to that mailbox. The TO address shows the original recipient and the FROM address shows the original sender. 

 

 

This allows the forwarded recipient to reply to the message and have it returned directly to the original sender. Messaging tracking also shows that the sender is the original sender and the recipient is the address configured for forwarding.

 

 

It is important to understand this behavior in scenarios where compliance or journaling is used as subsequent replies will bypass the original organization thereby bypassing journaling and compliance processes.

 

In this scenario the X-MS-Exchange-Organization-AutoForwarded header is not stamped on the message.

 

Autoforwarding and delivertoandforward:

Each mailbox also has a parameter called deliverToAndForward. This allows the administrator to specify if the message should also be delivered to the mailbox where forwarding is enabled. This setting applies ONLY to forwarding that is configured by using the forwardingAddress or forwardingSMTPAddress parameters. Client-side rules are unaware of this setting.

 

Autoforwarding and remoteDomains:

Within Exchange and Office 365, administrators can create remote domains. By default, every tenant leverages the ‘*’ domain. When a specific remote domain does not exist, the ‘*’ remote domain setting are applied to the message.

 

A property of a remote domain is autoForwardEnabled property. This allows administrators to define if auto-forwarding is allowed on messages destined to the domain specified.

 

PS C:> Get-RemoteDomain  | fl name,domainname,autoForwardEnabled

Name               : Default
DomainName         : *
AutoForwardEnabled : True

 

By default auto-forwarding is allowed to all domains. Administrators can use the remote domain settings to control how forwarding outside their organization is handled. Please note: this does not change how forwarding is handled within the organization.

 

In this example, I am creating a specific remote domain rule and have disabled auto-forwarding. 

 

PS C:> New-RemoteDomain -Name ExternalDomain -DomainName domain.com

Name                           DomainName                                   AllowedOOFType
—-                           ———-                                   ————–
ExternalDomain                 domain.com                                   External

 

PS C:> Set-RemoteDomain -Identity ExternalDomain -AutoForwardEnabled:$FALSE

 

The settings can be verified with get-remoteDomain:

 

PS C:> Get-RemoteDomain ExternalDomain | fl domainname,autoforwardenabled

DomainName         : domain.com
AutoForwardEnabled : False

 

To test this feature I set the forwardingAddress on the mailbox to a recipient in the domain.com domain. When a message is addressed to the mailbox where forwarding is enabled, it never makes it to the destination. The same behavior would occur if a client-side forwarding rule is used.

 

When a client-side rule or the forwardingSMTPAddress is used, the mail flow process stamps the X-MS-Exchange-Organization-AutoForwarded to TRUE. This header is not viewable on a message as the transport header firewall removes it. When the message is processed by the Transport service, if the remote domain specifically blocks auto-forwarding and the auto-forward header is present and set to TRUE, the message is turfed, and no NDR is generated. If no form of journaling is present and the deliverToAndForward setting of the mailbox is set to FALSE, the message is effectively lost.

 

When looking at message tracking we can see that the message failed to reach its destination. The message delivery status is NONE and no send events are noted.

 

 

 

Using the same remote domain, the mailbox is configured with a forwardingAddress instead. The mail contact used as the forwardingAddress object has an external email address in domain.com. When sending a message to the mailbox where forwarding is enabled, the mail is successfully delivered. This is also confirmed by message tracking.

 

 

In this example, the transport process does not use the X-MS-Exchange-Organization-AutoForwarded but instead adds the forwarded address as a recipient during a different stage of message processing. As the message traverses the transport stack it is not detected as a forwarded message and is therefore allowed to arrive at desired recipient.

 

Utilizing the forwardingAddress scenario is an excellent way for an administrator to bypass forwarding settings on remote domains while controlling when message forwarding is allowed.

 

We routinely receive requests on how to enable granular message forwarding or only allow specific users to use auto-forwarding capabilities. Unfortunately, with the forwarding settings being tied to remote domains it is very difficult to enable forwarding for only a subset of users or where granular forwarding on messages attributes is desired. To allow clients to create auto-forwarding rules would mean the remote domain would have to allow auto-forwarding for everyone. Administrators can attempt to discover and remove auto-forwarding rules using Get-InboxRule. 

 

In my experience, in most scenarios forwarding is disallowed on remote domains and enabling of forwarding is restricted to administrator configuration using the forwardingAddress parameter.

The Start-DatabaseAvailabilityGroup cmdlet is used to restore failed nodes in a Database Availability Group (DAG) by joining the evicted nodes back to the existing cluster. When a node joins a cluster it is assigned a unique decimal value ID.  For example, the first node is assigned Node ID 1, the second node Node ID 2, and so on up to 16 supported nodes in a DAG.  You can view the node ID assigned to a particular node via the registry.  Navigate to HKLM Cluster Nodes:

 

 

The subkeys are the decimal values assigned to each node.  Selecting a subkey will expose the NetBIOS node name assigned to the NodeID.  In this example Node ID 1 is assigned to server MBX-1.

 

 

The servers list of the DAG is a multi-valued attribute that represents the members of the DAG. 

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers

Name    : DAG
Servers : {MBX-4, MBX-3, MBX-2, MBX-1}

 

When Stop-DatabaseAvailabilityGroup is run, the task parses the servers list and determines if an action should be taken on that server. For example, if the next server on the list falls within the AD site being stopped, the cmdlet will take action on that server. 

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers,startedmailboxservers,stoppedmailboxservers

Name                  : DAG
Servers               : {MBX-4, MBX-3, MBX-2, MBX-1}
StartedMailboxServers : {MBX-3.domain.com, MBX-4.domain.com}
StoppedMailboxServers : {MBX-2.domain.com, MBX-1.domain.com}

 

After running Restore-DatabaseAvailabilityGroup, the nodes on the stopped servers list are evicted from the cluster.  This results in the Node IDs within the cluster registry associated with those nodes being freed. 

 

 

At this stage, the secondary site is now functional, independent of the original.  Hopefully there will be a time where the primary site is accessible and its nodes are ready to be added back to the cluster as functional members.  Start-DatabaseAvailabilityGroup is used to bring the nodes back to the cluster.

 

Occasionally when running Start-DatabaseAvailabilityGroup the following error is thrown:

 

WARNING: Server 'DAG-4' failed to be started as a member of database availability group 'DAG'. Error: A server-side database availability group administrative operation failed. Error: The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API’ "AddClusterNode() (MaxPercentage=37) failed with 0x13af. Error: The cluster node is not valid"' failed. [Server: DAG-1.domain.com]

 

D:UtilitiesERR>err 0x13af
# for hex 0x13af / decimal 5039
  ERROR_CLUSTER_INVALID_NODE                                     winerror.h
# The cluster node is not valid.
  SQL_5039_severity_16                                           sql_err
# MODIFY FILE failed. Specified size is less than current
# size.
# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x13af
# for hex 0x13af / decimal 5039
  ERROR_CLUSTER_INVALID_NODE                                     winerror.h
# The cluster node is not valid.
# 3 matches found for "0x13af"

 

Why does this error occur?

 

Most customers will never see this error, and those that do typically see this error only in testing, where the commands associated with the Datacenter Switchover process are executed in a quick sequential fashion.  When Start-DatabaseAvailabilityGroup is run the command steps through the stopped mailbox servers list.  If you look at the list carefully you will notice that the nodes most likely appear in reverse order that then they were originally added to the DAG.

 

[PS] C:>Get-DatabaseAvailabilityGroup DAG | fl name,servers,startedmailboxservers,stoppedmailboxservers

Name                  : DAG
Servers               : {MBX-4, MBX-3, MBX-2, MBX-1}
StartedMailboxServers : {MBX-3.domain.com, MBX-4.domain.com}
StoppedMailboxServers : {MBX-2.domain.com, MBX-1.domain.com}

In this example, MBX-1 was added before MBX-2.  This would indicate that MBX-1 was assigned Node ID 1 and MBX-2 was assigned Node ID 2.  As nodes are added to a cluster, the first available node ID is recycled. In this example, Start-DatabaseAvailabilityGroup first detects the server MBX-2, therefore this server is the first server attempted to add to the existing cluster.  The cluster service then attempts to assign Node ID 1 to MBX-2.  For a short period of time the cluster service caches the previous Node ID to Server mapping that originally existed.  Therefore, in cache, Node ID 1 is still assigned to server MBX-1.  In this instance though we are trying to add server MBX-2 and the cluster by default attempts to assign Node ID 1 – resulting in a collision between the desired Node ID assignment and what exists in cache.  This ultimately results in the invalid node ID error returned to the cmdlet.

 

In the case of using the ADsite parameter with Start-DatabaseAvailabilityGroup, adding MBX-2 fails while adding server MBX-1 succeeds.  Based on the previous explanation this is expected, the cmdlet failed to add MBX-2 and now moved to MBX-1.  MBX-1 during the join process is assigned the first available node ID, in this case Node ID 1, which matches what previously existed in cache.  Since this match existed, the join was successful.  Executing the command multiple times with the –ADSite parameter will eventually result in all nodes being successfully added to the DAG.  If you run Start-DatabaseAvailabilityGroup with the –mailboxServer parameter, the error will continue to be seen in the short term unless the server specified matches the first available node ID.

 

Again, most administrators will not encounter this error during production site activations.  The time between when Restore-DatabaseAvailablityGroup is run and the time that the primary site is available to be added back to the DAG is sufficient for the cache to expire.  In some circumstances it may become necessary to restart the Cluster service on the surviving nodes to force the cache to expire and correct this error.  You can also use Start-DatabaseAvailabilityGroup with the –mailboxServer parameter to add the nodes back in the original order they were installed.