(An Exchange 2010 version of this article can be found here: http://blogs.technet.com/b/timmcmic/archive/2010/05/30/network-port-design-and-exchange-2010-database-availability-groups.aspx)

A question that I often see come up is how many network ports should I have in my clustered nodes and how should I use them.

I generally see three different hardware configurations:

• Two network ports.
  • Usually 2 onboard or 1 onboard / 1 add-on.
• Three network ports
  • Usually 2 onboard / 1 add-on.
• Four network ports.
  • Usually 2 onboard / 2 add-on.

In some hardware there are now 4 port cards.  The information contained here can be expanded to include additional hardware / port configurations as they become available.

You’ll note that there is no configuration with a single network port – all clustered installations must at minimum have two network interfaces.  (Note:  VLANS to a single port are not two network interfaces).

When I’m advising customers the number of ports I recommend is dependant on the installation of Exchange. 

• Exchange 2007 SP1 SCC (Single Copy Cluster).
  • Generally recommend the 2 or 3 interface installation.
• Exchange 2007 SP1 CCR (Cluster Continuous Replication)
  • Generally recommend the 3 or 4 interface installation.

Using these recommendations I’ll break down their uses below using Windows 2008 clustering terminology.

Network Teaming

In the recommendations I’ll outline next you will see references to the use of network teaming.  It’s important to note that Microsoft does not support network teaming as this is hardware vendor supported and designed technology.  What it is though is a recognition that in absence of anyway to provide multiple client facing ports for Exchange network teaming does have a valid place in the overall high availability design.

When using network teaming, only the client facing network should be a teamed adapter and at all times the team created for NETWORK FAULT TOLERANCE.  Do not, for an Exchange instance, use any type of load balancing between ports.

For non-client facing networks it is not supported and not necessary to implement at network team (these would typically be your “heartbeat” networks).  (Refer to:  http://support.microsoft.com/kb/254101).  Windows clustering has the ability to balance and use all interfaces on the cluster designated for cluster use without the need to establish teaming for cluster / heartbeat communications.

From a support perspective any customer that establishes a teamed interface for the client side network should recognize that they may be asked to dissolve the team to support troubleshooting efforts.

Exchange 2007 SP1 SCC (Single Copy Cluster) – Two Network Ports

When using a single copy cluster with two network ports, your options are limited.  Consider the following design:

• First port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
• Second port set to “allow the cluster to use this network”.

Exchange 2007 SP1 SCC (Single Copy Cluster) – Three Network Ports

This option provides you with some additional flexibility and also allows you to mitigate issues on the client facing network.

• First and second port configured in a fault tolerate team set for “allow the cluster to use this network” and “allow clients to connect through this network”.
• Third port set for “allow the cluster to use this network”.

Exchange 2007 SP1 CCR (Cluster Continuous Replication) – Two Network Ports

When using a cluster continuous replication cluster with two network ports, your options are limited.  Consider the following design:

• First port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
• Second port set to “allow the cluster to use this network” and “allow clients to connect through this network”.

You’ll noticed that in this configuration both networks are set to “allow clients to connect through this network”.  This is necessary in order to establish the “private” network for use with log shipping functions.

To establish this network for log shipping functions, refer to the enable-continuousreplicationhostnames commandlet.  (http://technet.microsoft.com/en-us/library/bb690985.aspx / http://technet.microsoft.com/en-us/library/bb629629.aspx)

If used the replication service will first prefer to perform log shipping functions over the “private” network.  Should the private network be unavailable the replication service will resume log shipping functions over the “public” network.

Exchange 2007 SP1 CCR (Cluster Continuous Replication) – Three Network Ports

This option for CCR provides some additional flexibility.  We can use a minimum of three ports to assist us in mitigating two factors that affect the overall high availability of this solution.

• More then one adapter assigned to the client facing network.
• Establishing more then one path to replicate log files between the nodes.
• Allow log replication to first prefer to “private” network dedicating the primary function of the “public” network to servicing client requests.

Considering using these ports in the following manner:

• First and second port configured in a fault tolerant team set for “allow the cluster to use this network” and “allow clients to connect through this network”.
• Third port set for “allow the cluster to use this network” and “allow clients to connect through this network”.

You’ll noticed that in this configuration both networks are set to “allow clients to connect through this network”.  This is necessary in order to establish the “private” network for use with log shipping functions.

To establish this network for log shipping functions, refer to the enable-continuousreplicationhostnames commandlet.  (http://technet.microsoft.com/en-us/library/bb690985.aspx / http://technet.microsoft.com/en-us/library/bb629629.aspx)

If used the replication service will first prefer to perform log shipping functions over the “private” network.  Should the private network be unavailable the replication service will resume log shipping functions over the “public” network.

Exchange 2007 SP1 CCR (Cluster Continuous Replication) – Four Network Ports

This option provides even greater flexibility in terms of minimizing single points of failure and assisting to ensure log shipping functions can be successful.

• More then one adapter assigned to the client facing network.
• Establishing more then one path to replicate log files between the nodes.
• Allow log replication to first prefer to “private” network dedicating the primary function of the “public” network to servicing client requests.

I generally see the four port design implemented in two methods:

• Method #1:
  • First port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Second port set to “allow this cluster to use this network”.
  • Third port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Forth port set to :allow the cluster to use this network” and “allow clients to connect through this network”.
  • Continuous replication host names used with ports three and four networks.
• Method #2 (my personal preference):
  • First and second port configured in a network fault tolerant team set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Third port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Forth port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Continuous replication host names used with ports three and four networks.

Method 1 allows for individuals to not use network teaming on the public interface.  In this case the public interface is a single point of failure.  It does though allow for two secondary replication networks for log shipping functions providing three overall paths for replication service use.

Method 2, which is my personal preferred method, allows for individuals to establish a team on the public interface.  This provides high availability for the client facing network ports.  In addition, the remaining two interfaces are enabled for continuous replication host names.  This allows the replication service to have two secondary replication networks for log shipping functions providing three overall paths for replication service use.

You’ll noticed that in this configuration non-client facing networks are set to “allow clients to connect through this network”.  This is necessary in order to establish the “private” network for use with log shipping functions.

To establish this network for log shipping functions, refer to the enable-continuousreplicationhostnames commandlet.  (http://technet.microsoft.com/en-us/library/bb690985.aspx / http://technet.microsoft.com/en-us/library/bb629629.aspx)

If used the replication service will first prefer to perform log shipping functions over the “private” network.  Should the private network be unavailable the replication service will resume log shipping functions over the “public” network.

I hope this information was helpful as you consider the number of network ports for your Exchange clustered nodes.

*Updates

5/19/9 – Two port CCR scenario updated to include recommendation to use continuous replication host name for second port.

With Exchange 2007 Cluster Continuous Replication clusters the recommended quorum type for the host cluster is Majority Node Set with File Share Witness (Windows 2003) or Node Majority and File Share Witness (Windows 2008).

In this blog post I want to talk about two things that have an influence on these decisions.

(Note:  All information in this blog assumes a two node scenario since that is the maximum node count supported on Exchange 2007 CCR based clustered installations.) 

The first item is placement of the file share witness.

In order for a two node solution to maintain quorum, we have to have a minimum of two votes.  In our two node cluster scenarios we attempt to maintain two votes by locking the file share witness location.  When a node has the ability to establish an SMB file lock on the file share witness, that node gets the benefit of the vote.  The node that has the minimum two votes necessary has quorum, and will stay functional and host applications.  The node that has the remaining one vote is lost quorum, and will terminate its cluster service.

When both nodes are in the same data center the placement of the file share witness is generally not an issue.  When multiple data centers / physical locations are involved, where WAN connections are used to maintain connectivity between them, the placement of the file share witness is important.

In many scenarios customers are only dealing with a primary and secondary data centers.  Generally I would recommend that the file share witness would be placed in the location where Exchange will service user accounts.  In this case, if the link between the two nodes are down (for example – WAN failure), Exchange will stay functioning on the server where users will be serviced.  This is due to the fact that two votes are available in the primary data center so that node has quorum, and only one vote is available in the secondary data center and that node has lost quorum.  In the event that the primary data center is actually lost, and the secondary data center must be activated, users could follow the appropriate forceQuorum instructions for their operating system to force the solution online. 

Considerations with the aforementioned scenario is that when connectivity is lost between the two data centers Exchange stays functioning in the primary data center.  Manual activation of the secondary data center would be necessary in the event of full primary data center loss.  Should the active node in the primary data center stop functioning, the solution would still function using the node in the secondary data center and the file share witness in the primary data center.

Another scenario is where the file share witness is placed in the secondary data center.  When given the same WAN failure as outlined before, Exchange would automatically be moved to the node in the secondary data center since that is the only node that can maintain quorum (ie has two votes).  The node in the primary data center does not have access to the file share witness, and will terminate it’s cluster services (lost quorum).  This scenario does appeal to some.  For example, should the primary data center be lost Exchange would automatically come online in the secondary data center.  What I consider to be a drawback of this design is that any communications loss between the primary data center and the secondary data center would result in Exchange coming online only in the secondary data center automatically, and not being able to service users (assumes users use the same WAN connection between data centers).  As in the previous scenario, should the WAN be functioning and the node lost in the secondary data center, Exchange would function in the primary data center using the file share witness in the remote data center to maintain quorum.

The last scenario is for customers that have at least three data centers.  In this scenario, the assumption is that each data center has direct connectivity to each other (think triangle here).  For example, Node A would be placed in DataCenter1, Node B in DataCenter2, and the File Share Witness in DataCenter3.  Should DataCenter1 and DataCenter2 loss connectivity, each will have equal access to the file share witness.  The first to successfully lock the file share witness gets the benefit of the vote, and can maintain quorum.  Any node maintaining quorum in this scenario will continue to host existing applications, and arbitrate other applications from nodes that are lost quorum.

In the previous example you get automatic activation should either primary or secondary data center be unavailable, protection from a single WAN failure between any two datacenters, and automatic activation for any node failure.

In the first two examples above it is generally not relevant which node owns the cluster group.  The ability to lock the file share witness is derived from it’s placement on either side of the WAN and the ability to maintain that WAN connection.  It is in the three data center scenario that the location of the cluster group is of importance.  Let’s take a look at that…

The second item – which node owns the cluster group (Applies to Windows 2003 Only).

In Windows 2003 the cluster group contains the cluster name, cluster IP address, and majority node set resource (configured to use file share witness). 

If you review the private properties of the majority node set resource, you will see a timer value called MNSFileShareDelay.  (cluster <clusterFQDN> res “Majority Node Set” /priv)

Cluster.exe cluster-1.exchange.msft res “Majortiy Node Set” /priv

Listing private properties for ‘Majority Node Set’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  Majority Node Set    MNSFileShare                   \2003-DC1MNS_FSW_Cluster-1

D  Majority Node Set    MNSFileShareCheckInterval      240 (0xf0)

D  Majority Node Set    MNSFileShareDelay              4 (0x4)

By default the MNSFileShareDelay is 4 seconds.  You can configure this to a different value but in general this is not necessary. 

When there is a condition where the two member nodes cannot communicate, and there is a need to use the file share witness to maintain quorum, the node that owns the cluster group gets the first change to lock the file share witness.  The node that does not own the cluster group sleeps for MNSFileShareDelay – in this case 4 seconds.

The second item – which node owns the cluster group (Applies to Windows 2008 Only).

In Windows 2008 the cluster group is partially abstracted from the users.  The items that comprise the cluster group – ip address, network name, and quorum resource are now known as cluster core resources.

Like Windows 2003, Windows 2008 also implements a delay for nodes not owning the cluster core resources when attempting to lock the file share witness. 

If you review the private properties of the File Share Witness resource, you will see a value called ArbitrationDelay. 

Listing private properties for ‘File Share Witness (\HT-2MNS_FSW_MBX-1)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  File Share Witness   SharePath                      \HT-2MNS_FSW_MBX-1

   (\HT-2MNS_FSW_MBX-1) 

D  File Share Witness   ArbitrationDelay               6 (0x6)

   (\HT-2MNS_FSW_MBX-1)

The default arbitration delay value is 6 seconds and it is generally not necessary to change this value.

When there is a condition where the two member nodes (or greater since FSW can be used with more than two nodes in Windows 2008) can no longer communicate, and utilization of the file share witness is necessary in order to maintain quorum, the node that owns the cluster core resources gets the first attempt to lock the file share witness.  Challenging nodes will sleep for 6 seconds before attempting to lock the witness directory. 

So…why does this delay matter?

Take the example of the three data center scenario.  Datacenter1 hosts NodeA currently running a clustered mailbox server, Datacenter2 hosts NodeB currently running the cluster group, and DataCenter3 hosts the file share witness.  The link between DataCenter1 and DataCenter2 is interrupted, no interruption exists between DataCenter1 and DataCenter3 or DataCenter2 and DataCenter3 – all nodes have equal access to the file share witness.  Since the cluster group is owned on NodeB, NodeB will immediately lock the file share witness.  NodeA, since a lock already exists, will be unable to lock the file share witness and will terminate its cluster service.  NodeB will arbitrate the Exchange resources and bring them online.  Because of this delay, in the three location scenario, you may end up with results that were unexpected (for example, expecting NodeA to continue running Exchange without interruption).

When using the Exchange commandlets to manage cluster (move-clusteredmailboxserver) we do not take any actions in regards to the cluster group, we only act on the Exchange group.  Taking into account the above example, you might find it necessary to modify how you move the Exchange and cluster resources between nodes.  Let me give two examples of where you might modify how you move resources between nodes.

Example #1:  You have a three data center scenario outlined before.  Your primary client base accessing Exchange is in DataCenter1.  You have decided to run Exchange on NodeB in DataCenter2.  The cluster group remains on NodeA in DataCenter1.  The link between DataCenter1 and DataCenter2 is interrupted.  Connections from each data center to DataCenter3 are not impacted.  NodeA, which owns the cluster resources, is first to lock the file share witness.  NodeB, waiting it’s delay period, finds an existing lock and is unable to maintain quorum – the cluster service terminates.  NodeA successfully arbitrates the Exchange resources.  In this case by leaving the cluster group on the node in the main data center, when the link was lost Exchange came home so that user service could be continued.

Example #2:  You have the three data center scenario outlined before.  Your primary client base accessing Exchange is in DataCenter1.  It is time to apply patches to your operating system requiring a reboot.  You successfully apply the patches to NodeB in DataCenter2.  Post reboot, you issue a move command for Exchange resources (move-clusteredmailboxserver –identity <CMSNAME> –targetNode NodeB) and the resources move successfully.  You then patch NodeA and issue a reboot.  During the reboot process, the cluster automatically arbitrates the cluster group to NodeB.  When NodeA has completed rebooting, you issue a command to move the Exchange resources back to NodeA.  Sometime after these moves occur the link between DataCenter1 and DataCenter2 is interrupted.  The link between each data center and DataCenter3 is not impacted.  NodeB, currently owning the cluster group, is allowed first access to the file share witness and is successful in establishing a lock.  NodeA, which also has access, is unable to establish a lock and terminates its cluster service.  In this case Exchange is moved from NodeA to NodeB (and presumably users are now cutoff from mail services since the link between DataCenter1 and DataCenter2 is not available).

Example #3:  You have the three data center scenario outlined before.  Your primary client base accessing Exchange is in DataCenter1.  It is time to apply patches to your operating system requiring a reboot.  You successfully apply the patches to NodeB in DataCenter2.  Post reboot, you issue a move command for Exchange resources (move-clusteredmailboxserver –identity <CMSNAME> –targetNode NodeB) and the resources move successfully.  You then patch NodeA and issue a reboot.  During the reboot process, the cluster automatically arbitrates the cluster group to NodeB.  When NodeA has completed rebooting, you issue a command to move the Exchange resources back to NodeA.  You also issue a command to move the cluster group back to NodeA (presumably because you’ve read and understood this blog).  (Cluster <clusterFQDN> group “Cluster Group” /moveto:<NODE>).  Sometime after these moves occur the link between DataCenter1 and DataCenter2 is interrupted.  The link between each data center and DataCenter3 is not impacted.  NodeA, currently owning the cluster group, is allowed first access to the file share witness and is successful in establishing a lock.  NodeB, which also has access, is unable to establish a lock and terminates its cluster service.  In this case Exchange is not impacted.

In most installations I work on it is not necessary to manage the cluster group – both nodes are located at the same location with the file share witness in the same location as the nodes.  If using multiple data centers, consider what is outlined here in the management of your Exchange and cluster resources.

===========================================================

***SEE UPDATE***

===========================================================

With the enhancements in Windows 2008 to allow for multi-subnet clustering it is becoming more common to see this utilized with Exchange 2007 SP1 installations. 

When implementing a clustered solution, it is a requirement that there be a minimum of two interfaces on each node, and that each node can maintain communications across those interfaces.  I see administrators implement this requirement in two different fashions with multi-subnet clusters:

• The “public” interface of each node resides in different subnets with the “private” interfaces residing in a stretched subnet.
• The “public” interface of each node resides in different subnets with the “private” interfaces also residing in different subnets.

If you are the second bullet, you’ll want to continue reading this blog.  (If you are the first bullet you’ll probably want to read it anyway since you’ve made it this far…)

For users that have a configuration where both network interfaces are in different subnets this will generally require routing between those two subnets.  A common mis-configuration that I see in this design is the use of default gateways on both of these network interfaces.

When a user attempts to configure two network interfaces each with a default gateway, the following error is noted from the operating system:

The text in this message is specifically important as it highlights at this time that this configuration will not produce the desired results.

The most likely cluster configuration where Exchange is used, with this type of clustering, is cluster continuous replication (CCR).  When multiple default gateways are defined, users may see inconsistent results in the performance and ability to replicate logs between the nodes.  The replication issues between nodes are also exacerbated when continuous replication hostnames are used utilizing the secondary networks with the default gateway assigned.  These issues are secondary to any issues that the cluster service many have maintaining communications between the nodes and any communications issues clients may have connecting to the nodes.

If the default gateways are removed from the “private” adapters, reliable routed communications can only occur over the “public” interface.  So…if two default gateways cannot be used, how should we ensure proper communications over both the “public” interface and “private” interface where both reside in different routed subnets.

The first part of this solution is to ensure that the binding order of the network interfaces is set correctly in the operating system.  To confirm the binding order:

• Open the network connections control panel.
• Choose the advanced menu (if menu is disabled, enable it by selecting Organize –> Layout –> Menu Bar).
• Select advanced settings from the advanced menu.
• On the adapters and bindings tab, ensure that the “public” interface is first in the list, with all secondary interfaces following after.

The second part of the solution is to maintain the default gateway on the “public” interface.

The third part of the solution is to enable persistent static routes on the “private” interfaces.  In terms of the routes we simple need to configure routes to other “private” networks using gateway addresses that have the ability to route between those “private” networks.  All other traffic not matching this route should be handled by the default gateway of the “public” adapter.

Let’s take a look at an example. 

I desire to have a two node Exchange 2007 SP1 CCR cluster on Windows 2008 with each node residing in a different subnet.

NodeA:

Public

• IP Address 192.168.0.100
• Subnet Mask 255.255.255.0
• Default Gateway 192.168.0.254

Private

• IP Address 10.0.0.1
• Subnet Mask 255.255.255.0
• Gateway on network 10.0.0.254

NodeB:

Public:

• IP Address 192.168.1.100
• Subnet Mask 255.255.255.0
• Default Gateway 192.168.1.254

Private

• IP Address 10.0.1.1
• Subnet Mask 255.255.255.0
• Gateway on network 10.0.1.254

(Note that gateway on network is not the default gateway setting but is the gateway on the private interface network that can route packets to the private network on the other nodes.)

In this case I would want to establish the necessary persistent static routes on each node.  In order to accomplish this, I can use the route add command.  The structure of the route command:

NodeA:  Route add 10.0.1.0 mask 255.255.255.0 10.0.0.254 –p

NodeB:  Route add 10.0.0.0 mask 255.255.255.0 10.0.1.254 –p

The –p switch will ensure that the routes are persistent lasting after a reboot.  Failure to use the –p will result in the routes being removed post a reboot operation. 

You can verify that the routes are correct by running route print and reviewing the persistent route information.

By utilizing only a default gateway on the “public” adapter, and static routes on the “private” adapters, you can ensure safe routed paths for client communications, cluster communications, and replication service log shipping.

========================================================

Update – 1-18-2010

========================================================

With Windows 2008 and Windows 2008 R2 the recommendation to manage static routes has changed.  Although route add should work the management of routes has technically been replaced with functionality in netsh.  Therefore, it is a recommendation that the netsh commands be utilized to implement and manage static routes.

I will leave the previous information un-edited in the blog since many people have used it.

The first step in implementing static routes with the netsh command is to determine the interface names.  The interface name is the logical name assigned to the network connection – for example Local Area Connection 1.  It is recommended that these networks be named into something more logical, for example LAN-Replication-A.  The logical network names may be the same on all nodes.

You can also determine that adapter name from an ipconfig /all.  (Note the name listed below in RED)

Windows IP Configuration

   Host Name . . . . . . . . . . . . : DAG-1
   Primary Dns Suffix  . . . . . . . : exchange.msft
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : exchange.msft

Ethernet adapter LAN:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Adapter
   Physical Address. . . . . . . . . : 00-15-5D-00-02-07
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::dd27:d7f6:549f:6b9b%11(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.0.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.0.2(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.0.254
   DHCPv6 IAID . . . . . . . . . . . : 234886493
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-45-7C-F8-00-15-5D-00-02-07
   DNS Servers . . . . . . . . . . . : 192.168.0.253
                                       192.168.0.252
                                       192.168.0.251
   Primary WINS Server . . . . . . . : 192.168.0.253
   Secondary WINS Server . . . . . . : 192.168.0.252
                                       192.168.0.251
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter LAN-Replication-A:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Adapter #2
   Physical Address. . . . . . . . . : 00-15-5D-00-02-08
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.0.0.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   NetBIOS over Tcpip. . . . . . . . : Disabled

The netsh command format to add static routes looks like:

netsh interface ipv4 add route <IP/Mask> “InterfaceName” Gateway

Using the information from the above example, the following netsh commands would be utilized in place of route add:

NodeA:  netsh interface ipv4 add route 10.0.1.0/24 “LAN-Replication-A” 10.0.0.254

NodeB:  netsh interface ipv4 add route 10.0.0.0/24 “LAN-Replication-A” 10.0.1.254

The netsh command automatically assumes – unless otherwise specified in the command – that the route added is persistent.

If the command completes successfully the route addition can be verified by running:

netsh interface ipv4 show route

The following is sample output with the added route in RED (output truncated to show sample line including prefix and gateway):

C:>netsh interface ip show route

Prefix                    Idx  Gateway/Interface Name

————————  —  ————————

10.0.1.0/24                11  LAN-Replication-A

This is how the netsh command can be used to accomplish what would have previously been done with route add.

========================================================

========================================================

Update 9/18/2012:

Updated the netsh verification command to show correct syntax.

========================================================

There are times where cluster setup operations fail during the active clustered mailbox server installation.

Depending on where the failure occurred, when reviewing the resources in cluster administrator, it may appear that setup has fully completed.   For example – there is an IP, Name, System Attendant, Information Store, and Database Instance resources.  When a setup failure occurs, all or some of these resources are in a offline or failed state.  Generally issuing an online command will bring the resources online leading people to believe that the installation was “successful”.

A clustered installation, regardless of resource state in the cluster, is never successful until the /newCMS or /recoverCMS command completes successfully.  When creating your cluster, always allow the /newCMS command or /recoverCMS command to complete successfully – troubleshooting any failures that you have – so that you have a fully supported, fully installed clustered solution.

This question has seem to come up a lot lately, and since I have an opinion on it I figured it’s time to blog it.

When you run Exchange setup for the first time we will always create a default storage group and default mailbox database (also depending on installation order you may get a second storage group with a public folder database).  This default database serves several purposes:

• In order for the information store service to start there must be at least one storage group and one mailbox database defined.
• The system attendant mailbox defined for the server is automatically stored in the first mailbox database created during setup.
• Each storage group has a system mailbox.

Here is a sample dump of a system mailbox showing homeMDB stamped.

Expanding base ‘CN=SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928},CN=Microsoft Exchange System Objects,DC=exchange,DC=msft’…
Getting 1 entries:
Dn: CN=SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928},CN=Microsoft Exchange System Objects,DC=exchange,DC=msft
cn: SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928};
deliveryMechanism: 0;
displayName: SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928};
distinguishedName: CN=SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928},CN=Microsoft Exchange System Objects,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = (  );
homeMDB: CN=2008-MBX3-SG1-DB1,CN=2008-MBX3-SG1,CN=InformationStore,CN=2008-MBX3,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
instanceType: 0x4 = ( WRITE );
legacyExchangeDN: /o=Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928};
mail: SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928}@exchange.msft;
mailNickname: SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928};
msExchHideFromAddressLists: TRUE;
msExchHomeServerName: /o=Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=2008-MBX3;
msExchMailboxGuid: 4ab4dda2-166e-42c1-9ab7-951919825c39;
msExchMailboxSecurityDescriptor: O:SYG:SYD:(A;CI;CCDCRC;;;SY);
name: SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928};
objectCategory: CN=ms-Exch-System-Mailbox,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (2): top; msExchSystemMailbox;
objectGUID: 4ab4dda2-166e-42c1-9ab7-951919825c39;
proxyAddresses: SMTP:SystemMailbox{00608BD1-A3C2-4F33-8499-AA68EFE80928}@exchange.msft;
uSNChanged: 41160;
uSNCreated: 41159;
whenChanged: 9/16/2008 10:32:19 AM Eastern Daylight Time;
whenCreated: 9/16/2008 10:32:19 AM Eastern Daylight Time;

———–

Here is a sample LDP dump of the system attendant mailbox showing homeMDB stamped.

Expanding base ‘CN=Microsoft System Attendant,CN=2008-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft’…
Getting 1 entries:
Dn: CN=Microsoft System Attendant,CN=2008-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: System Attendant;
cn: Microsoft System Attendant;
deletedItemFlags: 5;
deliveryMechanism: 0;
delivExtContTypes: <ldp: Binary blob 8 bytes>;
displayName: Microsoft System Attendant;
distinguishedName: CN=Microsoft System Attendant,CN=2008-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = (  );
garbageCollPeriod: 0;
homeMDB: CN=2008-MBX1-SG1-DB1,CN=2008-MBX1-SG1,CN=InformationStore,CN=2008-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
homeMTA: CN=Microsoft MTA,CN=2008-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Exchange,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
instanceType: 0x4 = ( WRITE );
legacyExchangeDN: /o=Exchange/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=2008-MBX1/cn=Microsoft System Attendant;
mail: 2008-MBX1-SA@exchange.msft;
mailNickname: 2008-MBX1-SA;
mDBOverHardQuotaLimit: 100000;
mDBUseDefaults: FALSE;
msExchMailboxSecurityDescriptor: O:LAG:LAD:(A;;CCRC;;;SY);
msExchPoliciesIncluded: {7E47E963-8F11-496A-A63E-34521AB66DFF},{26491CFC-9E50-4857-861B-0CB8DF22B5D7};
name: Microsoft System Attendant;
objectCategory: CN=ms-Exch-Exchange-Admin-Service,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (2): top; exchangeAdminService;
objectGUID: d6c9d7c9-c7e9-4990-812b-8a246a4e9d0f;
proxyAddresses: SMTP:2008-MBX1-SA@exchange.msft;
showInAdvancedViewOnly: TRUE;
uSNChanged: 33319;
uSNCreated: 33257;
whenChanged: 9/15/2008 6:16:16 PM Eastern Daylight Time;
whenCreated: 9/15/2008 6:10:15 PM Eastern Daylight Time;

———–

In terms of my opinion, the second two bullet points are the most important.

When the default mailbox database and/or storage group are removed and recreated, the homeMDB associated with the system mailbox and system attendant mailbox are no longer valid.  When a storage group and database are recreated, the system mailbox should be created and the system attendant mailbox rehomed to this store.  The issue here is when these operations fail.  When any of these operations fail, the following operations may also fail:

• Move mailbox may no longer function.
• OWA free busy publishing may no longer function.
• Event sinks may not function correctly if the system mailbox is absent.

In most cases the rehoming and recreation procedures work without issues, but why chance it?

In many cases this query arises from customers that desire to script installations of Exchange.  In my opinion it’s just as easy to script the deletion and recreation of the default storage group and mailbox database as it is is to update the names to your standard naming convention, and move the paths.

In order to change the name of an existing storage group, use the set-storagegroup command.

set-storagegroup –identity <ServerFirst Storage Group> –name <NewName>

In order to change the name of an existing database, use the set-mailboxdatabase (or set-publicfolderdatabase).

set-mailboxdatabase –identity <ServerMailbox Database> –name <NewName>

In order to move the paths, use the move commandlets.

move-storagegrouppath –identity <ServerNewName> –logFolderPath <path> –systemFolderPath <path> –configurationOnly:$TRUE

move-databasepath –identity <ServerNewName> –edbFilePath <pathfile.edb> –configurationOnly:$True  (This is where you’ll want to specify the default naming convention for your edb file).

In each of these commands you will notice that I use the –configurationOnly switch.  Using this switch should be a safe operation since the database / storage group being modified should not contain any user mailboxes.  By utilizing the configurationOnly switch, I can run the same series of commands regardless of the installation type for Exchange (standalone / single copy cluster / cluster continuous replication).

On a standalone server or single copy cluster installation, when I use the above command with –configurationOnly, I will be prompted to mount a blank database.  If a yes is provided to the mount command, a new log stream will be created at the logFolderPath location, a new checkpoint file created at the systemPathLocation, and a new edb file created at the edbFilePath with the edb name specified in the command.  No existing files will be moved or retained.  If I choose not to run the command with –configurationOnly, the files will be automatically moved to their new paths for me and the existing files preserved.

(Note:  If using a single copy cluster installation do not forget to update the database / storage group dependencies to include all lettered volumes and mountpoints where Exchange data resides for that database / storage group.)

On a clustered continuous replication installation, when I use the above command with –configurationOnly, I will be prompted to mount a blank database.  If a yes is provided to the mount command, a new log stream will be created at the logFolderPath location, a new checkpoint file created at the systemPathLocation, and a new edb file created at the edbFilePath with the edb name specified in the command.  No existin files will be moved or retained.  If I choose not to run the command with –configurationOnly, an error will result indcating that moves on CCR members can only be performed with –configurationOnly. 

In the case of CCR clusters that have no user data to retain, it is safe to use the –configurationOnly switch and mount the blank database.  The replication service will be smart enough to detect the path change, start replicating log files to the passive node, and replay the logs and build the database at it’s new location.  If there is user data to retain, the files should be manually moved to their new locations on both nodes prior to mounting the database.

(Note:  If standby continuous replication is already enabled on any database, and existing data is preserved either by automatically moving files or manually moving files to their new paths, the same files will need to be manually moved on the SCR target machine.  If mounting a blank database into a new log stream, the replication service on SCR is smart enough to detect the path change, being replicating log files, and rebuild the database in their new location.)

If following these steps you should have successfully renamed the default storage group and database to the naming convention of the organization.  The log files, checkpoint files, and databases files should be moved to their desired location with the edb file name matching the organizations naming convention.  The added bonus comes in the fact that we preserved the homeMDB thereby preventing the need to re-create the system mailbox or rehome the system attendant mailbox.

Good luck and happy renaming!