Recently there have been some questions about transaction log rolling and continuous replication.  In some cases these questions often surround storage group copy status showing an initializing state (http://blogs.technet.com/timmcmic/archive/2009/01/26/get-storagegroupcopystatus-initializing.aspx).

Under normal circumstances, the only time that log would roll, is when we’ve reached a log full condition.  If the server is being utilized, this is not a problem, as logs will roll naturally as the server processes activity.

There are times though where the server is relatively idle.  This would mean the current log generation would not receive enough transaction activity against it to cause it to roll over.  This is where “transaction log roll” is important.  If the current log file (ENN.log) contains a durable (or hard) commit, and that log is not filled in a period of time, it will be rolled over and shipped to the other side.  (This is not an immediate process, if we rolled a log over every time there was a durable (hard) commit we’d generate a ton of logs).  The article referenced above gives examples of how to calculate the time that a log would roll over should it contain a durable (hard) commit.  The article above also contains the following text highlighting this behavior:

“The log roll mechanism does not generate transaction logs in the absence of user or other database activity. In fact, log roll is designed to occur only when there is a partially filled log.”

This information is important to us for several reasons.

The first is generally if logs roll why do my storage groups stay initializing for hours at a time.  The answer is because the current log does not contain a durable commit.  If you were to restart the replication service or suspend and resume a replication instance manually the first replication state you will encounter is initializing.  We remain in initializing until a log is generated, copied, inspected, and put out for replay with divergence information determined.  If no durable (hard) commit exists in the source log stream, the logs may not be rolled over until there is a durable (hard) commit or user activity, which means replication would stay in an initializing state for a while.  My suggestion is, if this is a test environment, simply send mail / dismount the source databases / etc.  In production, I’ve seen people script email to test mailboxes at a schedule time with a test mailbox located in each database.  This causes a durable commit, which will eventually result in log file roll over and shipment to the other side.

The second reason is that log file roll can cause churn in the log file stream which does not appear normal.  If you reference the link above you can see that an idle storage group could generate up to 960 log files a day.  This is especially true of the storage group contains some type of system mailboxes (which exchange accesses causing a durable commit) or test mailboxes which the user is accessing.  In either scenario, there may not be enough load by either process to force log roll to occur naturally, so Exchange rolls the log for you at a certain time.  This causes some concern, especially when looking at the log file drive on a test server etc and questioning why so many logs were generated.  IE – there wasn’t enough traffic to generate 960 megs of logs, which is probably correct, but there was enough traffic to put a durable commit into each of those 960 logs such that we rolled and shipped them without being full in attempts to keep both sides up to date.

The third reason I pointed this out is that there seems to be confusion on when log roll should occur.  This leads to people believing the log roll should occur no matter what, when as indicated it should only occur if the log contains a durable (hard) commit. 

There are other operations besides user activity or a durable (hard) commit which will cause the current transaction log to roll:

• An attachment record is created in a log when a database is mounted.
• A VSS backup occurs of the active node.
• A VSS backup occurs of the passive node.
• An online streaming backup occurs of the active node.

I hope everyone finds this information helpful.

Recently there was a lively internal debate regarding how to use restore-storagegroupcopy and the –force switch.

The documentation regarding the restore-storagegroupcopy command can be found at http://technet.microsoft.com/en-us/library/aa996024.aspx.

According to the TechNet documentation:

“The Force parameter can be used when the task is run programmatically and prompting for administrative input is inappropriate. If Force is not provided in the cmdlet, administrative input is prompted. If Force is provided in the cmdlet, but the value is omitted, its default value is $true. When the Restore-StorageGroupCopy cmdlet is run to make an SCR target viable for mounting, the Force parameter must be included when the SCR source is not available.”

You’ll notice in this text that –force is required for standby continuous replication when the SCR source is not available.

So the first question is what constitutes the source being unavailable.  In the most general terms the source is unavailable when the shares where the log files reside are not available such that the restore-storagegroupcopy command can be run and the remaining logs copied between machines.

For Windows 2003 based sources, and Windows 2008 non-shared storage clusters, the shares are generally not available when the entire machine is offline.  For Windows 2008 shared storage clusters, the shares may not be available because their corresponding file server resources are offline in the clustered mailbox server group (for example, a stop-clusteredmailboxserver was issued taking the entire CMS offline, including the file server resources).  Of course there are other reasons that shares may not be available, like network issues / misc hardware issues / etc.

The reason I point this out is that if the source is available, and the –force command is being used, we will not copy the delta logs over to the SCR source and mark the databases mountable.  This effectively causes the database mount process to fail indicating log files necessary for recovery are not present.  Manual recovery using eseutil /r /a would have to be performed in order for the databases to mount.

The second question is how can I overcome this limitation so this does not happen to me?  The answer to that is simple.  If you run the restore-storagegroupcopy without the –force we will attempt to copy delta logs.  Should the source be unavailable, we will fail the copy procedure with a meaningful message indicating that the delta logs cannot be copied, and –force is necessary.  After receiving this error you can repeat the restore-storagegroupcopy, this time specifying the –force.  Since –force was required, the logs will not be copied (source unavailable) but the databases will be marked mountable.

Rule of Thumb:  First try restore-storagegroupcopy and only run restore-storagegroupcopy –force if indicated to do so in the error text of the command.

===========================================================

Example of successful activation using restore-storagegroupcopy where the shares are available (no –force used).

===========================================================

Environment:  Source cluster / target standalone.

The source clustered mailbox server was stopped using stop-clusteredmailboxserver.

An eseutil /ml of the source log directory was run, the end of the log file can be seen here.  You will see that the log stream is complete through the E01.log.

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000070.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000071.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000072.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000073.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000074.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000075.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000076.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 14.921 seconds.

Prior to running the restore-storagegroupcopy an eseutil /ml was run against the logs on the SCR target.  You will note that the same logs are present with the exception of the E01.log.  (This is expected, even when the source CMS is shutdown gracefully the last log in the series is not copied to the SCR target.)

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000070.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000071.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000072.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000073.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000074.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000075.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000076.log – OK

No damaged log files were found.

Operation completed successfully in 7.63 seconds.

At this time the shares on the source are available, and the mailbox stores dismounted.  A restore-storagegroupcopy –standbymachine <machine> is run and completes without error.  The following events are noted in the application log.

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/30/2009 8:20:16 AM
Event ID:      2114
Task Category: Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The replication instance for storage group MBX-2MBX-2-SG2 has started copying transaction log files. The first log file successfully copied was generation 119.

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/30/2009 8:20:16 AM
Event ID:      2085
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The Restore-StorageGroupCopy operation on MBX-2MBX-2-SG2 was successful. All logs were successfully copied.

I then re-ran the eseutil /ml against the log series.  You will note that after the restore-storagegroupcopy –standbymachine:<machine> that the e01.log is now present, it was successfully copied as a part of the restore process.

I followed up with an eseutil /ml of the target log directory, you can now see that the E01.log is present in the directory.

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000071.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000072.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000073.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000074.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000075.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000076.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 7.250 seconds.

The last operation was to mount the databases.  At this time the databases mounted successfully – eseutil /r /a was not required.

Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          4/30/2009 8:25:06 AM
Event ID:      9523
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The Microsoft Exchange Database "MBX-3-SG2MBX-3-SG2-DB1" has been started.

Database File: G:MBX-2MBX-2-SG2-DatabaseMBX-2-SG2-DB1.edb
Transaction Logfiles: F:MBX-2MBX-2-SG2-Logs
Base Name (logfile prefix): E01
System Path: E:MBX-2MBX-2-SG2-System

===========================================================

===========================================================

Example of successful activation using restore-storagegroupcopy where the shares are not available (-force used).

===========================================================

Environment:  Source cluster / target standalone.

The clustered nodes comprising the source solution were completely shutdown making them completely unavailable.

Prior to shutting the nodes down, after issuing a stop-clusteredmailboxserver, and eseutil /ml was run against the log directory.  You will see the log stream is complete through E01.log.

Verifying log files…
     Base name: e01

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000092.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000093.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000094.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 0.78 seconds

Prior to running the restore-storagegroupcopy, an eseutil /ml was run against the logs on the SCR target.  You will note that the E01.log is not present.

Verifying log files…
     Base name: e01

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000092.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000093.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000094.log – OK

No damaged log files were found.

Operation completed successfully in 0.64 seconds.

At this time a restore-storagegroupcopy –standbymachine <MACHINE> was issued.  The following error was noted and expected since the source is no longer available.

[PS] G:>Restore-StorageGroupCopy -Identity MBX-2MBX-2-SG2 -StandbyMachine MBX-3
Restore-StorageGroupCopy : Restore failed to verify if the database on ‘MBX-2’ is mounted. Verify that the database is dismounted and then use the -Force parameter to restore the storage group copy.
At line:1 char:25
+ Restore-StorageGroupCopy  <<<< -Identity MBX-2MBX-2-SG2 -StandbyMachine MBX-3

After receiving an error that –force was necessary, the command was re-run using restore-storagegroupcopy –standbymachine –force.  The following information was presented in the Exchange Management Shell window:

[PS] G:>Restore-StorageGroupCopy -Identity MBX-2MBX-2-SG2 -StandbyMachine MBX-3    -force
WARNING: Performing a Restore-StorageGroupCopy operation on storage group
‘MBX-2-SG2’ with the Force option. Data loss is expected for this storage group.

The following events were noted in the application log:

Log Name:      Application
Source:        MSExchangeRepl
Date:          5/3/2009 10:37:39 AM
Event ID:      2139
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The forced Restore-StorageGroupCopy operation on MBX-2MBX-2-SG2 was successful. However, there may be some data loss.

After the command complete successfully, an eseutil /ml was performed against the log stream.  You will note that the e01.log is not present in the target log directory, since the remaining logs could not be copied due to the SCR source being unavailable.

Verifying log files…
     Base name: e01

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000092.log – OK 
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000093.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000094.log – OK

No damaged log files were found.

Operation completed successfully in 0.64 seconds.

At this time the database was successfully mounted as indicated by the following event in the application log.

Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          5/3/2009 10:44:06 AM
Event ID:      9523
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The Microsoft Exchange Database "MBX-3-SG2MBX-3-SG2-DB1" has been started.

Database File: G:MBX-2MBX-2-SG2-DatabaseMBX-2-SG2-DB1.edb
Transaction Logfiles: F:MBX-2MBX-2-SG2-Logs
Base Name (logfile prefix): E01
System Path: E:MBX-2MBX-2-SG2-System

===========================================================

===========================================================

Example of successful activation using restore-storagegroupcopy where the shares are available (-force used).

===========================================================

Environment:  Source cluster / target standalone.

The source clustered mailbox server was stopped using stop-clusteredmailboxserver.

An eseutil /ml of the source log directory was run, the end of the log file can be seen here.  You will see that the log stream is complete through the E01.log.

      Log file: F:MBX-2MBX-2-SG2-LogsE010000007A.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007B.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007C.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007D.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007E.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 16.219 seconds.

Prior to running the restore-storagegroupcopy an eseutil /ml was run against the logs on the SCR target.  You will note that the same logs are present with the exception of the E01.log.  (This is expected, even when the source CMS is shutdown gracefully the last log in the series is not copied to the SCR target.)

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000079.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007A.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007B.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007C.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007D.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007E.log – OK

No damaged log files were found.

Operation completed successfully in 0.359 seconds.

At this time a restore-storagegroupcopy with the –force command was run.  Please note:  The source shares are available so –force is NOT NECESSARY.  Here is sample Exchange Management Shell output.

[PS] C:WindowsSystem32>Restore-StorageGroupCopy -Identity MBX-2MBX-2-SG2 –StandbyMachine MBX-3 –force

WARNING: Performing a Restore-StorageGroupCopy operation on storage group
‘MBX-2-SG2’ with the Force option. Data loss is expected for this storage
group.

The command completed successfully as indicated by returning to the Exchange Management Shell prompt without error.  The following event was noted in the application log.

Log Name:      Application
Source:        MSExchangeRepl
Date:          5/1/2009 8:29:41 AM
Event ID:      2139
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The forced Restore-StorageGroupCopy operation on MBX-2MBX-2-SG2 was successful. However, there may be some data loss.

As follow up eseutil /ml was run against the logs on the SCR target machine.  You will note that the E01.log was not copied even though the restore-storagegroupcopy –force command completed successfully.

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000077.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000078.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000079.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007A.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007B.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007C.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007D.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007E.log – OK

No damaged log files were found.

Operation completed successfully in 0.187 seconds.

At this time a database mount attempt was performed, and failed with the following events noted in the application log.

Log Name:      Application
Source:        MSExchangeIS
Date:          5/1/2009 8:32:13 AM
Event ID:      9518
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
Error Current log file missing starting Storage Group /DC=com/DC=domain/DC=domain/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=Organization/CN=Administrative Groups/CN=Exchange Administrative Group (FYDIBOHF23SPDLT)/CN=Servers/CN=MBX-3/CN=InformationStore/CN=MBX-3-SG2 on the Microsoft Exchange Information Store.
Storage Group – Initialization of Jet failed.

Log Name:      Application
Source:        ESE
Date:          5/1/2009 8:32:13 AM
Event ID:      455
Task Category: Logging/Recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
MSExchangeIS (2984) MBX-3-SG2: Error -1811 (0xfffff8ed) occurred while opening logfile f:MBX-2MBX-2-SG2-LogsE01.log.

The –1811 error translates to:

# for decimal -1811 / hex 0xfffff8ed
  JET_errFileNotFound
# /* File not found */
  JET_errFileNotFound
# /* File not found */
  JET_errFileNotFound
# /* File not found */
# 3 matches found for "-1811"

In this case the –force command was improperly used resulting in logs not being copied to the SCR target.  The databases could be mounted if they were manually recovered using eseutil /r /a or the logs manually copied to the SCR target.

This behavior is BY DESIGN.  The –force command does not check to see if the SCR source is available, therefore no log file copy attempts are made.

===========================================================

There maybe time in either Windows 2003 or Windows 2008 where it may become necessary to evict a clustered node that has Exchange 2007 installed on it.

Under normal circumstances evicting a clustered node is a benign procedure.  When the node has Exchange 2007 installed on it special precautions must be taken.

When Exchange 2007 is installed on a clustered node a special DLL (exres.dll) is registered for the cluster service.  This dll contains the extensions in cluster that define the system attendant, information store, and databases instance clustered resources.  You can see the resource definitions in the cluster registry hive (HKLM –> System –> Cluster –> ResourceTypes).

If you select one of the Exchange resource types, you will see that the DLL that defines it (DLLName) is exres.dll.

The resource types that are registered in a cluster are local to each node.  When a node is evicted from a cluster, the local configuration is destroyed.  If the node is joined back to an existing cluster, the Exchange resource types are no longer registered.  This will effectively prevent this node from participating in the cluster.

In terms of Exchange there is no manual way to re-register the cluster extensions.  Exchange 2007 does not have a reinstall procedure.  If you attempt to rerun setup for the passive mailbox role, an error is generated indicating the role is already installed (because technically it is).  In some cases you are able to uninstall the mailbox role successfully, where the uninstall is not successful though there are no manual removal steps that can be used.  The worse case scenario is that the entire operating system must be rebuilt in order to facilitate installing Exchange.

To avoid this, use the following steps to successfully remove Exchange to facilitate evicting a clustered node:

1)  Run setup.com /mode:uninstall /roles:mt,mb

(Note:  MT is necessary to remove the management tools.  By default, any role install also includes the management tools.  By default, any uninstall only applies to the role specific – to have a complete removal you must specify both the mailbox role and management tools role.)

2)  Evict the node from the cluster.

3)  Re-join the node to the cluster.

4)  Run setup.com /mode:install /roles:mailbox to re-establish the passive node mailbox role installation.

Occasionally I have an opportunity to work with our Exchange MVPs.  Neil Hobson, one of our Exchange MVPs, has recently started a series on how to test SCR in a production environment. 

If you have the time I would suggest checking it out!

Part 1:  http://www.msexchange.org/articles_tutorials/exchange-server-2007/high-availability-recovery/testing-scr-production-environment-part1.html

Part 2:  http://www.msexchange.org/articles_tutorials/exchange-server-2007/high-availability-recovery/testing-scr-production-environment-part2.html

Look for parts 3 and 4 in the upcoming weeks.

Tim

When running ExBPA against a Windows 2008 cluster (either CCR or SCC) the following best practices may be flagged:

• “Cluster dedicated heartbeat network priority set incorrectly” – http://technet.microsoft.com/en-us/library/aa997088.aspx
• “Dedicated cluster heartbeat not found” – http://technet.microsoft.com/en-us/library/aa998726.aspx
• “Quorum log too small” – http://technet.microsoft.com/en-us/library/aa995830.aspx

These rules only apply to Windows 2003 clusters and the recommendations should not be followed for Exchange 2007 / Windows 2008 clusters.

This is scheduled to be corrected in a future ExBPA rules update.

As many probably already know Microsoft has announced the next version of Exchange – Exchange Server 2010.

Since I like to write about high availability, I thought I’d let you know of two things that are no longer present in Exchange 2010 from a high availability standpoint.

• Local Continuous Replication
• Single Copy Clusters

In Exchange 2010 the high availability landscape has changed to something known as the Database Availability Group. 

I’ll write more about Exchange 2010 later and as things progress…but keep this information in mind as you think about your HA and Exchange 2010 deployments.

(An Exchange 2010 version of this article can be found here: http://blogs.technet.com/b/timmcmic/archive/2010/05/30/network-port-design-and-exchange-2010-database-availability-groups.aspx)

A question that I often see come up is how many network ports should I have in my clustered nodes and how should I use them.

I generally see three different hardware configurations:

• Two network ports.
  • Usually 2 onboard or 1 onboard / 1 add-on.
• Three network ports
  • Usually 2 onboard / 1 add-on.
• Four network ports.
  • Usually 2 onboard / 2 add-on.

In some hardware there are now 4 port cards.  The information contained here can be expanded to include additional hardware / port configurations as they become available.

You’ll note that there is no configuration with a single network port – all clustered installations must at minimum have two network interfaces.  (Note:  VLANS to a single port are not two network interfaces).

When I’m advising customers the number of ports I recommend is dependant on the installation of Exchange. 

• Exchange 2007 SP1 SCC (Single Copy Cluster).
  • Generally recommend the 2 or 3 interface installation.
• Exchange 2007 SP1 CCR (Cluster Continuous Replication)
  • Generally recommend the 3 or 4 interface installation.

Using these recommendations I’ll break down their uses below using Windows 2008 clustering terminology.

Network Teaming

In the recommendations I’ll outline next you will see references to the use of network teaming.  It’s important to note that Microsoft does not support network teaming as this is hardware vendor supported and designed technology.  What it is though is a recognition that in absence of anyway to provide multiple client facing ports for Exchange network teaming does have a valid place in the overall high availability design.

When using network teaming, only the client facing network should be a teamed adapter and at all times the team created for NETWORK FAULT TOLERANCE.  Do not, for an Exchange instance, use any type of load balancing between ports.

For non-client facing networks it is not supported and not necessary to implement at network team (these would typically be your “heartbeat” networks).  (Refer to:  http://support.microsoft.com/kb/254101).  Windows clustering has the ability to balance and use all interfaces on the cluster designated for cluster use without the need to establish teaming for cluster / heartbeat communications.

From a support perspective any customer that establishes a teamed interface for the client side network should recognize that they may be asked to dissolve the team to support troubleshooting efforts.

Exchange 2007 SP1 SCC (Single Copy Cluster) – Two Network Ports

When using a single copy cluster with two network ports, your options are limited.  Consider the following design:

• First port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
• Second port set to “allow the cluster to use this network”.

Exchange 2007 SP1 SCC (Single Copy Cluster) – Three Network Ports

This option provides you with some additional flexibility and also allows you to mitigate issues on the client facing network.

• First and second port configured in a fault tolerate team set for “allow the cluster to use this network” and “allow clients to connect through this network”.
• Third port set for “allow the cluster to use this network”.

Exchange 2007 SP1 CCR (Cluster Continuous Replication) – Two Network Ports

When using a cluster continuous replication cluster with two network ports, your options are limited.  Consider the following design:

• First port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
• Second port set to “allow the cluster to use this network” and “allow clients to connect through this network”.

You’ll noticed that in this configuration both networks are set to “allow clients to connect through this network”.  This is necessary in order to establish the “private” network for use with log shipping functions.

To establish this network for log shipping functions, refer to the enable-continuousreplicationhostnames commandlet.  (http://technet.microsoft.com/en-us/library/bb690985.aspx / http://technet.microsoft.com/en-us/library/bb629629.aspx)

If used the replication service will first prefer to perform log shipping functions over the “private” network.  Should the private network be unavailable the replication service will resume log shipping functions over the “public” network.

Exchange 2007 SP1 CCR (Cluster Continuous Replication) – Three Network Ports

This option for CCR provides some additional flexibility.  We can use a minimum of three ports to assist us in mitigating two factors that affect the overall high availability of this solution.

• More then one adapter assigned to the client facing network.
• Establishing more then one path to replicate log files between the nodes.
• Allow log replication to first prefer to “private” network dedicating the primary function of the “public” network to servicing client requests.

Considering using these ports in the following manner:

• First and second port configured in a fault tolerant team set for “allow the cluster to use this network” and “allow clients to connect through this network”.
• Third port set for “allow the cluster to use this network” and “allow clients to connect through this network”.

You’ll noticed that in this configuration both networks are set to “allow clients to connect through this network”.  This is necessary in order to establish the “private” network for use with log shipping functions.

To establish this network for log shipping functions, refer to the enable-continuousreplicationhostnames commandlet.  (http://technet.microsoft.com/en-us/library/bb690985.aspx / http://technet.microsoft.com/en-us/library/bb629629.aspx)

If used the replication service will first prefer to perform log shipping functions over the “private” network.  Should the private network be unavailable the replication service will resume log shipping functions over the “public” network.

Exchange 2007 SP1 CCR (Cluster Continuous Replication) – Four Network Ports

This option provides even greater flexibility in terms of minimizing single points of failure and assisting to ensure log shipping functions can be successful.

• More then one adapter assigned to the client facing network.
• Establishing more then one path to replicate log files between the nodes.
• Allow log replication to first prefer to “private” network dedicating the primary function of the “public” network to servicing client requests.

I generally see the four port design implemented in two methods:

• Method #1:
  • First port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Second port set to “allow this cluster to use this network”.
  • Third port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Forth port set to :allow the cluster to use this network” and “allow clients to connect through this network”.
  • Continuous replication host names used with ports three and four networks.
• Method #2 (my personal preference):
  • First and second port configured in a network fault tolerant team set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Third port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Forth port set to “allow the cluster to use this network” and “allow clients to connect through this network”.
  • Continuous replication host names used with ports three and four networks.

Method 1 allows for individuals to not use network teaming on the public interface.  In this case the public interface is a single point of failure.  It does though allow for two secondary replication networks for log shipping functions providing three overall paths for replication service use.

Method 2, which is my personal preferred method, allows for individuals to establish a team on the public interface.  This provides high availability for the client facing network ports.  In addition, the remaining two interfaces are enabled for continuous replication host names.  This allows the replication service to have two secondary replication networks for log shipping functions providing three overall paths for replication service use.

You’ll noticed that in this configuration non-client facing networks are set to “allow clients to connect through this network”.  This is necessary in order to establish the “private” network for use with log shipping functions.

To establish this network for log shipping functions, refer to the enable-continuousreplicationhostnames commandlet.  (http://technet.microsoft.com/en-us/library/bb690985.aspx / http://technet.microsoft.com/en-us/library/bb629629.aspx)

If used the replication service will first prefer to perform log shipping functions over the “private” network.  Should the private network be unavailable the replication service will resume log shipping functions over the “public” network.

I hope this information was helpful as you consider the number of network ports for your Exchange clustered nodes.

*Updates

5/19/9 – Two port CCR scenario updated to include recommendation to use continuous replication host name for second port.

With Exchange 2007 Cluster Continuous Replication clusters the recommended quorum type for the host cluster is Majority Node Set with File Share Witness (Windows 2003) or Node Majority and File Share Witness (Windows 2008).

In this blog post I want to talk about two things that have an influence on these decisions.

(Note:  All information in this blog assumes a two node scenario since that is the maximum node count supported on Exchange 2007 CCR based clustered installations.) 

The first item is placement of the file share witness.

In order for a two node solution to maintain quorum, we have to have a minimum of two votes.  In our two node cluster scenarios we attempt to maintain two votes by locking the file share witness location.  When a node has the ability to establish an SMB file lock on the file share witness, that node gets the benefit of the vote.  The node that has the minimum two votes necessary has quorum, and will stay functional and host applications.  The node that has the remaining one vote is lost quorum, and will terminate its cluster service.

When both nodes are in the same data center the placement of the file share witness is generally not an issue.  When multiple data centers / physical locations are involved, where WAN connections are used to maintain connectivity between them, the placement of the file share witness is important.

In many scenarios customers are only dealing with a primary and secondary data centers.  Generally I would recommend that the file share witness would be placed in the location where Exchange will service user accounts.  In this case, if the link between the two nodes are down (for example – WAN failure), Exchange will stay functioning on the server where users will be serviced.  This is due to the fact that two votes are available in the primary data center so that node has quorum, and only one vote is available in the secondary data center and that node has lost quorum.  In the event that the primary data center is actually lost, and the secondary data center must be activated, users could follow the appropriate forceQuorum instructions for their operating system to force the solution online. 

Considerations with the aforementioned scenario is that when connectivity is lost between the two data centers Exchange stays functioning in the primary data center.  Manual activation of the secondary data center would be necessary in the event of full primary data center loss.  Should the active node in the primary data center stop functioning, the solution would still function using the node in the secondary data center and the file share witness in the primary data center.

Another scenario is where the file share witness is placed in the secondary data center.  When given the same WAN failure as outlined before, Exchange would automatically be moved to the node in the secondary data center since that is the only node that can maintain quorum (ie has two votes).  The node in the primary data center does not have access to the file share witness, and will terminate it’s cluster services (lost quorum).  This scenario does appeal to some.  For example, should the primary data center be lost Exchange would automatically come online in the secondary data center.  What I consider to be a drawback of this design is that any communications loss between the primary data center and the secondary data center would result in Exchange coming online only in the secondary data center automatically, and not being able to service users (assumes users use the same WAN connection between data centers).  As in the previous scenario, should the WAN be functioning and the node lost in the secondary data center, Exchange would function in the primary data center using the file share witness in the remote data center to maintain quorum.

The last scenario is for customers that have at least three data centers.  In this scenario, the assumption is that each data center has direct connectivity to each other (think triangle here).  For example, Node A would be placed in DataCenter1, Node B in DataCenter2, and the File Share Witness in DataCenter3.  Should DataCenter1 and DataCenter2 loss connectivity, each will have equal access to the file share witness.  The first to successfully lock the file share witness gets the benefit of the vote, and can maintain quorum.  Any node maintaining quorum in this scenario will continue to host existing applications, and arbitrate other applications from nodes that are lost quorum.

In the previous example you get automatic activation should either primary or secondary data center be unavailable, protection from a single WAN failure between any two datacenters, and automatic activation for any node failure.

In the first two examples above it is generally not relevant which node owns the cluster group.  The ability to lock the file share witness is derived from it’s placement on either side of the WAN and the ability to maintain that WAN connection.  It is in the three data center scenario that the location of the cluster group is of importance.  Let’s take a look at that…

The second item – which node owns the cluster group (Applies to Windows 2003 Only).

In Windows 2003 the cluster group contains the cluster name, cluster IP address, and majority node set resource (configured to use file share witness). 

If you review the private properties of the majority node set resource, you will see a timer value called MNSFileShareDelay.  (cluster <clusterFQDN> res “Majority Node Set” /priv)

Cluster.exe cluster-1.exchange.msft res “Majortiy Node Set” /priv

Listing private properties for ‘Majority Node Set’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  Majority Node Set    MNSFileShare                   \2003-DC1MNS_FSW_Cluster-1

D  Majority Node Set    MNSFileShareCheckInterval      240 (0xf0)

D  Majority Node Set    MNSFileShareDelay              4 (0x4)

By default the MNSFileShareDelay is 4 seconds.  You can configure this to a different value but in general this is not necessary. 

When there is a condition where the two member nodes cannot communicate, and there is a need to use the file share witness to maintain quorum, the node that owns the cluster group gets the first change to lock the file share witness.  The node that does not own the cluster group sleeps for MNSFileShareDelay – in this case 4 seconds.

The second item – which node owns the cluster group (Applies to Windows 2008 Only).

In Windows 2008 the cluster group is partially abstracted from the users.  The items that comprise the cluster group – ip address, network name, and quorum resource are now known as cluster core resources.

Like Windows 2003, Windows 2008 also implements a delay for nodes not owning the cluster core resources when attempting to lock the file share witness. 

If you review the private properties of the File Share Witness resource, you will see a value called ArbitrationDelay. 

Listing private properties for ‘File Share Witness (\HT-2MNS_FSW_MBX-1)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  File Share Witness   SharePath                      \HT-2MNS_FSW_MBX-1

   (\HT-2MNS_FSW_MBX-1) 

D  File Share Witness   ArbitrationDelay               6 (0x6)

   (\HT-2MNS_FSW_MBX-1)

The default arbitration delay value is 6 seconds and it is generally not necessary to change this value.

When there is a condition where the two member nodes (or greater since FSW can be used with more than two nodes in Windows 2008) can no longer communicate, and utilization of the file share witness is necessary in order to maintain quorum, the node that owns the cluster core resources gets the first attempt to lock the file share witness.  Challenging nodes will sleep for 6 seconds before attempting to lock the witness directory. 

So…why does this delay matter?

Take the example of the three data center scenario.  Datacenter1 hosts NodeA currently running a clustered mailbox server, Datacenter2 hosts NodeB currently running the cluster group, and DataCenter3 hosts the file share witness.  The link between DataCenter1 and DataCenter2 is interrupted, no interruption exists between DataCenter1 and DataCenter3 or DataCenter2 and DataCenter3 – all nodes have equal access to the file share witness.  Since the cluster group is owned on NodeB, NodeB will immediately lock the file share witness.  NodeA, since a lock already exists, will be unable to lock the file share witness and will terminate its cluster service.  NodeB will arbitrate the Exchange resources and bring them online.  Because of this delay, in the three location scenario, you may end up with results that were unexpected (for example, expecting NodeA to continue running Exchange without interruption).

When using the Exchange commandlets to manage cluster (move-clusteredmailboxserver) we do not take any actions in regards to the cluster group, we only act on the Exchange group.  Taking into account the above example, you might find it necessary to modify how you move the Exchange and cluster resources between nodes.  Let me give two examples of where you might modify how you move resources between nodes.

Example #1:  You have a three data center scenario outlined before.  Your primary client base accessing Exchange is in DataCenter1.  You have decided to run Exchange on NodeB in DataCenter2.  The cluster group remains on NodeA in DataCenter1.  The link between DataCenter1 and DataCenter2 is interrupted.  Connections from each data center to DataCenter3 are not impacted.  NodeA, which owns the cluster resources, is first to lock the file share witness.  NodeB, waiting it’s delay period, finds an existing lock and is unable to maintain quorum – the cluster service terminates.  NodeA successfully arbitrates the Exchange resources.  In this case by leaving the cluster group on the node in the main data center, when the link was lost Exchange came home so that user service could be continued.

Example #2:  You have the three data center scenario outlined before.  Your primary client base accessing Exchange is in DataCenter1.  It is time to apply patches to your operating system requiring a reboot.  You successfully apply the patches to NodeB in DataCenter2.  Post reboot, you issue a move command for Exchange resources (move-clusteredmailboxserver –identity <CMSNAME> –targetNode NodeB) and the resources move successfully.  You then patch NodeA and issue a reboot.  During the reboot process, the cluster automatically arbitrates the cluster group to NodeB.  When NodeA has completed rebooting, you issue a command to move the Exchange resources back to NodeA.  Sometime after these moves occur the link between DataCenter1 and DataCenter2 is interrupted.  The link between each data center and DataCenter3 is not impacted.  NodeB, currently owning the cluster group, is allowed first access to the file share witness and is successful in establishing a lock.  NodeA, which also has access, is unable to establish a lock and terminates its cluster service.  In this case Exchange is moved from NodeA to NodeB (and presumably users are now cutoff from mail services since the link between DataCenter1 and DataCenter2 is not available).

Example #3:  You have the three data center scenario outlined before.  Your primary client base accessing Exchange is in DataCenter1.  It is time to apply patches to your operating system requiring a reboot.  You successfully apply the patches to NodeB in DataCenter2.  Post reboot, you issue a move command for Exchange resources (move-clusteredmailboxserver –identity <CMSNAME> –targetNode NodeB) and the resources move successfully.  You then patch NodeA and issue a reboot.  During the reboot process, the cluster automatically arbitrates the cluster group to NodeB.  When NodeA has completed rebooting, you issue a command to move the Exchange resources back to NodeA.  You also issue a command to move the cluster group back to NodeA (presumably because you’ve read and understood this blog).  (Cluster <clusterFQDN> group “Cluster Group” /moveto:<NODE>).  Sometime after these moves occur the link between DataCenter1 and DataCenter2 is interrupted.  The link between each data center and DataCenter3 is not impacted.  NodeA, currently owning the cluster group, is allowed first access to the file share witness and is successful in establishing a lock.  NodeB, which also has access, is unable to establish a lock and terminates its cluster service.  In this case Exchange is not impacted.

In most installations I work on it is not necessary to manage the cluster group – both nodes are located at the same location with the file share witness in the same location as the nodes.  If using multiple data centers, consider what is outlined here in the management of your Exchange and cluster resources.

===========================================================

***SEE UPDATE***

===========================================================

With the enhancements in Windows 2008 to allow for multi-subnet clustering it is becoming more common to see this utilized with Exchange 2007 SP1 installations. 

When implementing a clustered solution, it is a requirement that there be a minimum of two interfaces on each node, and that each node can maintain communications across those interfaces.  I see administrators implement this requirement in two different fashions with multi-subnet clusters:

• The “public” interface of each node resides in different subnets with the “private” interfaces residing in a stretched subnet.
• The “public” interface of each node resides in different subnets with the “private” interfaces also residing in different subnets.

If you are the second bullet, you’ll want to continue reading this blog.  (If you are the first bullet you’ll probably want to read it anyway since you’ve made it this far…)

For users that have a configuration where both network interfaces are in different subnets this will generally require routing between those two subnets.  A common mis-configuration that I see in this design is the use of default gateways on both of these network interfaces.

When a user attempts to configure two network interfaces each with a default gateway, the following error is noted from the operating system:

The text in this message is specifically important as it highlights at this time that this configuration will not produce the desired results.

The most likely cluster configuration where Exchange is used, with this type of clustering, is cluster continuous replication (CCR).  When multiple default gateways are defined, users may see inconsistent results in the performance and ability to replicate logs between the nodes.  The replication issues between nodes are also exacerbated when continuous replication hostnames are used utilizing the secondary networks with the default gateway assigned.  These issues are secondary to any issues that the cluster service many have maintaining communications between the nodes and any communications issues clients may have connecting to the nodes.

If the default gateways are removed from the “private” adapters, reliable routed communications can only occur over the “public” interface.  So…if two default gateways cannot be used, how should we ensure proper communications over both the “public” interface and “private” interface where both reside in different routed subnets.

The first part of this solution is to ensure that the binding order of the network interfaces is set correctly in the operating system.  To confirm the binding order:

• Open the network connections control panel.
• Choose the advanced menu (if menu is disabled, enable it by selecting Organize –> Layout –> Menu Bar).
• Select advanced settings from the advanced menu.
• On the adapters and bindings tab, ensure that the “public” interface is first in the list, with all secondary interfaces following after.

The second part of the solution is to maintain the default gateway on the “public” interface.

The third part of the solution is to enable persistent static routes on the “private” interfaces.  In terms of the routes we simple need to configure routes to other “private” networks using gateway addresses that have the ability to route between those “private” networks.  All other traffic not matching this route should be handled by the default gateway of the “public” adapter.

Let’s take a look at an example. 

I desire to have a two node Exchange 2007 SP1 CCR cluster on Windows 2008 with each node residing in a different subnet.

NodeA:

Public

• IP Address 192.168.0.100
• Subnet Mask 255.255.255.0
• Default Gateway 192.168.0.254

Private

• IP Address 10.0.0.1
• Subnet Mask 255.255.255.0
• Gateway on network 10.0.0.254

NodeB:

Public:

• IP Address 192.168.1.100
• Subnet Mask 255.255.255.0
• Default Gateway 192.168.1.254

Private

• IP Address 10.0.1.1
• Subnet Mask 255.255.255.0
• Gateway on network 10.0.1.254

(Note that gateway on network is not the default gateway setting but is the gateway on the private interface network that can route packets to the private network on the other nodes.)

In this case I would want to establish the necessary persistent static routes on each node.  In order to accomplish this, I can use the route add command.  The structure of the route command:

NodeA:  Route add 10.0.1.0 mask 255.255.255.0 10.0.0.254 –p

NodeB:  Route add 10.0.0.0 mask 255.255.255.0 10.0.1.254 –p

The –p switch will ensure that the routes are persistent lasting after a reboot.  Failure to use the –p will result in the routes being removed post a reboot operation. 

You can verify that the routes are correct by running route print and reviewing the persistent route information.

By utilizing only a default gateway on the “public” adapter, and static routes on the “private” adapters, you can ensure safe routed paths for client communications, cluster communications, and replication service log shipping.

========================================================

Update – 1-18-2010

========================================================

With Windows 2008 and Windows 2008 R2 the recommendation to manage static routes has changed.  Although route add should work the management of routes has technically been replaced with functionality in netsh.  Therefore, it is a recommendation that the netsh commands be utilized to implement and manage static routes.

I will leave the previous information un-edited in the blog since many people have used it.

The first step in implementing static routes with the netsh command is to determine the interface names.  The interface name is the logical name assigned to the network connection – for example Local Area Connection 1.  It is recommended that these networks be named into something more logical, for example LAN-Replication-A.  The logical network names may be the same on all nodes.

You can also determine that adapter name from an ipconfig /all.  (Note the name listed below in RED)

Windows IP Configuration

   Host Name . . . . . . . . . . . . : DAG-1
   Primary Dns Suffix  . . . . . . . : exchange.msft
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : exchange.msft

Ethernet adapter LAN:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Adapter
   Physical Address. . . . . . . . . : 00-15-5D-00-02-07
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::dd27:d7f6:549f:6b9b%11(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.0.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.0.2(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.0.254
   DHCPv6 IAID . . . . . . . . . . . : 234886493
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-12-45-7C-F8-00-15-5D-00-02-07
   DNS Servers . . . . . . . . . . . : 192.168.0.253
                                       192.168.0.252
                                       192.168.0.251
   Primary WINS Server . . . . . . . : 192.168.0.253
   Secondary WINS Server . . . . . . : 192.168.0.252
                                       192.168.0.251
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter LAN-Replication-A:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft Virtual Machine Bus Network Adapter #2
   Physical Address. . . . . . . . . : 00-15-5D-00-02-08
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 10.0.0.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   NetBIOS over Tcpip. . . . . . . . : Disabled

The netsh command format to add static routes looks like:

netsh interface ipv4 add route <IP/Mask> “InterfaceName” Gateway

Using the information from the above example, the following netsh commands would be utilized in place of route add:

NodeA:  netsh interface ipv4 add route 10.0.1.0/24 “LAN-Replication-A” 10.0.0.254

NodeB:  netsh interface ipv4 add route 10.0.0.0/24 “LAN-Replication-A” 10.0.1.254

The netsh command automatically assumes – unless otherwise specified in the command – that the route added is persistent.

If the command completes successfully the route addition can be verified by running:

netsh interface ipv4 show route

The following is sample output with the added route in RED (output truncated to show sample line including prefix and gateway):

C:>netsh interface ip show route

Prefix                    Idx  Gateway/Interface Name

————————  —  ————————

10.0.1.0/24                11  LAN-Replication-A

This is how the netsh command can be used to accomplish what would have previously been done with route add.

========================================================

========================================================

Update 9/18/2012:

Updated the netsh verification command to show correct syntax.

========================================================

There are times where cluster setup operations fail during the active clustered mailbox server installation.

Depending on where the failure occurred, when reviewing the resources in cluster administrator, it may appear that setup has fully completed.   For example – there is an IP, Name, System Attendant, Information Store, and Database Instance resources.  When a setup failure occurs, all or some of these resources are in a offline or failed state.  Generally issuing an online command will bring the resources online leading people to believe that the installation was “successful”.

A clustered installation, regardless of resource state in the cluster, is never successful until the /newCMS or /recoverCMS command completes successfully.  When creating your cluster, always allow the /newCMS command or /recoverCMS command to complete successfully – troubleshooting any failures that you have – so that you have a fully supported, fully installed clustered solution.