Great blog post on installing rollup updates on DAG members. 

http://www.ucblogs.net/blogs/exchange/archive/2009/12/15/Installing-Exchange-2010-rollups-on-DAG-servers.aspx

Recently one of our Exchange MVPs posted a question to our support engineers…Is it necessary to move the cluster group / cluster core resources in Exchange 2007 / Exchange 2010?

As a reminder…the cluster core resources are generally the cluster name and cluster IP address.  This is the management name specified when the cluster is created.

In Windows 2003 this would be known as the “Cluster Group” and would contain the cluster name, cluster IP address, and majority node set or quorum disk resource.

In Windows 2008 and Windows 2008 R2 this would be known as the “Cluster Core Resources” (the actual group in cluster though is still called the “Cluster Group”) and generally contains the cluster name, cluster IP address, and file share witness or quorum disk resource. 

In Windows 2003 the cluster group is visible and manageable in cluster administrator.  In Windows 2008 you can view the cluster core resources, but management requires using the cluster.exe command line (or powershell in Windows 2008 R2).

So…enough background and back to the question…

From the platforms perspective it should not be necessary to maintain either of these group or move the cluster core resources when rebooting nodes etc.  When the resources are owned on a node where the cluster service is stopping, the resources will automatically be arbitrated to another node in the cluster. 

I personally though do not prefer to take this approach.  I prefer to know that the resources have been moved off the node that I’m managing and have been successfully arbitrated to another node in the cluster.

Therefore, I recommend the following…

Prior to managing a node verify whether or not the node owns the cluster core resources.

Run the following command – cluster.exe <FQDNCluster> group

C:>cluster dag.exchange.msft group
Listing status for all available resource groups:

Group                Node            Status
——————– ————— ——
Cluster Group        DAG-3           Online
Available Storage    DAG-1           Offline

As you can see from the output the “Cluster Group” currently resides on DAG-3 and is online.

If DAG-3 is the node to be rebooted, the cluster group can be moved to another node.

Run the following command – cluster.exe <FQDN> group “Cluster Group” /moveto:<NODE>

Here is an example of where the resources did not arbitrate successfully.  You can see that the group successfully moved to the specified node, but the status is partially online.

C:>cluster.exe dag.exchange.msft group "Cluster Group" /moveto:DAG-1

Moving resource group ‘Cluster Group’…

Group                Node            Status
——————– ————— ——
Cluster Group        DAG-1           Partially Online

Any other status but online should be investigated and corrected prior to rebooting or managing the cluster services on the nodes.

Here is an example of a successful move.

C:>cluster.exe dag.Exchange.msft group "Cluster Group" /moveto:DAG-1

Moving resource group ‘Cluster Group’…

Group                Node            Status
——————– ————— ——
Cluster Group        DAG-1           Online

If the group successfully moved, the node information should display the name of the node where the group was moved and the status should show online.

In Exchange 2010 when using the Database Availability Group (DAG) we leverage the cluster services in Windows 2008 and Windows 2008 R2. 

When utilizing the cluster services in Windows 2008 and Windows 2008 R2 the cluster core resources – cluster name is a Kerberos enabled name.  This requires that a machine account be created within the directory for association with this cluster name resource.  This is known as the CNO or cluster name object. 

With previous versions of Exchange this Kerberos enabled machine account was created in the domain leveraging the credentials / rights assigned to the user logged into the machine using the failover cluster management tools to establish the clustered services.  This would require either the user’s account have permissions to create and enable machine accounts within the domain or the machine account to be pre-staged with the appropriate rights assigned.

In Exchange 2010 the establishment of the cluster and the cluster name object is performed by using either the Exchange Management Console or Exchange Management Shell.  Specifically the cluster creation occurs when running the command add-databaseavailabilitygroupserver when adding the first server to the database availability group.  Since we are leveraging the Exchange management tools to establish the cluster service, this also means that we are utilizing RBAC.  In this case when the first server is added to the DAG, the remote powershell contacts the replication service of the first node to be added.  In turn, the replication service of that local machine installs the failover cluster management service and begins cluster creation utilizing the credentials assigned to LOCAL SYSTEM (since the replication service starts on the machine in the LOCAL SYSTEM security context).  Therefore, the individual user rights to create machine accounts in the domain are not leveraged when establishing the clustered services for the database availability group, but rather that of the replication service.

In environments where computer account creation is restricted, it may become necessary to pre-stage the CNO for the clustered services and assign the appropriate rights.  There are two methods which work to establish this security context:

1)  Assign the machine account of the first node added to the DAG with full control of the pre-staged object.

2)  Assign the Exchange Trusted Subsystem universal security group with full control of the pre-staged object.

The first method works by ensuring that the LOCAL SYSTEM security context will be able to manage the pre-staged computer account fully.  The second method works because the Exchange Trusted Subsystem group contains the machine accounts of all Exchange servers in the domain.  If utilizing method one you need to ensure that the first member added to the DAG is the machine granted permissions.  If the second method is utilized any DAG member can be added as all members have rights to manage the account.

Here is an example of how to pre-stage the machine account utilizing the local system rights of the first DAG member.

1)  Select the appropriate container in Active Directory where you wish the account to be created.  Right click, select NEW –> COMPUTER.

2)  In the Computer Name field, type the name that was assigned to the database availability group.

3)  After the account is created, right click on the account and select properties.  Select the security tab.  (Note:  You may have to enable advanced features in Active Directory Users and Computers).

4)  Select the ADD button to present the add dialog.

5)  Select the Object Types button.  De-select all options and select the COMPUTERS option.  Press OK.

6)   In the Enter the object name to select field, type the name of the first DAG member.  Select Check Name to ensure the name resolves to a valid account in the directory.  Select OK to add the account.

7)   Select the machine account in the Group or User Names field, assign the machine account full control.  Press OK.

8)  Locate the account in the container where it was created.  Right click on the account and select disable account.

9)  When presented with the disable dialog, select YES.

10)  After the object has been successfully disabled,  press the OK button on the success confirmation.

11)  At this point allow time for active directory replication to occur.  Once replication has completed, the first database availability group member can be added and the cluster services established.

======================================

Update 3/14/2010

======================================

It was recently asked how can I programmatically achieve establishing these permissions?  Thanks to Jeff Kizner and Robert Gillies for providing these instructions.

cd ad:

$comp = Get-ADComputer DAG01

$sid = (Get-ADGroup "Exchange Trusted Subsystem").sid

$rights = [System.DirectoryServices.ActiveDirectoryRights]::GenericAll

$perm = [System.Security.AccessControl.AccessControlType]::Allow

$acl = get-acl $comp

$ace = new-object System.DirectoryServices.ActiveDirectoryAccessRule $sid, $rights, $perm

$acl.AddAccessRule($ace)

set-acl -AclObject $acl $comp

======================================

The official TechNet documentation for establishing a Windows 2003 / Exchange 2007 SP1 cluster can be found at: 

• Exchange 2007 SP1 Single Copy Clusters (SCC) – http://technet.microsoft.com/en-us/library/bb125217.aspx
• Exchange 2007 SP1 Cluster Continuous Replication (CCR) – http://technet.microsoft.com/en-us/library/bb124521.aspx
• Installing Exchange 2007 SP1 Cluster Continuous Replication (CCR) on Windows 2003 – http://technet.microsoft.com/en-us/library/aa997144.aspx
• Installing Exchange 2007 SP1 Single Copy Clusters (SCC) on Windows 2003 – http://technet.microsoft.com/en-us/library/bb124899.aspx

What I want to do in this blog post is outline an alternate method for installing the cluster different from the instructions referenced above for Windows 2003 clustered nodes.

The first recommendation that I make when establishing the cluster service is to not use the GUI mode setup included with Exchange 2007.  My main rational for this advice is that when GUI mode setup fails you generally have to revert to command line setup to correct conditions and repeat the setup operations.  With this in mind it’s much easier to start with the command line setup and continue through.

In each of the links above you will see that the basic structure of a command line setup for the active roles installation is:

setup.com /mode:install /roles:mailbox /newCMS /cmsName /cmsIPAddress [/css] [/cmsDataPath]

The passive role installation is:

setup.com /mode:install /roles:mailbox

The way to think about the active role command is the combination of the passive role (setup.com /mode:install /roles:mailbox) with the establishment of the clustered services (setup.com /newCMS /cmsName /cmsIPAddress [/css] [/cmsDataPath]).

These commands do not have to be run together.  With this in mind my second recommendation is to establish the passive role first on each clustered node.  I recommend this for a few reasons:

• By separating the commands we can identify failures and address them easier.
• This method uses a foundational approach whereby the nodes are established and the cluster services verified, the mailbox roles established and verified, and only after that the clustered instance established and verified.
• When the commands are combined, and the failure occurs in the /newCMS portion of the command, further setup attempts using the same command error indicating the mailbox roles is currently installed and must be uninstalled.  Although the error is true, uninstalling the mailbox role and rerunning the same command without correcting the conditions that failed the cluster portion of setup will generally result is hitting the same error again (and thus the loop continues).
• In a shared storage environment the restart of the clustered services during mailbox role installation causes storage to move between nodes.  Setup attempts to move this storage back, but is sometimes not successful in doing so.  This causes the /newCMS portion of the setup command to fail, and thus the active role command to fail.  Separating the commands allows us to establish storage and arbitrate it to the “active” node for setup operations after the mailbox role has been installed.
• A setup code error exists where when an existing CMS exists in a cluster, and the passive node installation is run, the passive nodes machine account is promoted to full control permissions of the CMS.  This results in errors in the management console and management shell that an account was found with full administrator rights that is not an Exchange administrator.  (Note:  The error is truly benign, I just point it out because it is annoying.)

Now I will outline for you the steps that I use when establishing a Windows 2003 / Exchange 2007 SP1 Cluster.  Please refer to these additional blog posts for some other helpful information regarding these installations:

• http://blogs.technet.com/timmcmic/archive/2009/03/16/exchange-2007-newcms-or-recovercms-fails-when-installing-on-windows-2003-clusters.aspx
• http://blogs.technet.com/timmcmic/archive/2009/04/15/one-thing-not-to-do-when-exchange-2007-cluster-setup-fails.aspx
• http://blogs.technet.com/timmcmic/archive/2009/04/12/exchange-2007-renaming-the-default-storage-group-mailbox-database-or-deleting-and-recreating.aspx
• http://blogs.technet.com/timmcmic/archive/2009/04/26/file-share-witness-fsw-placement-and-the-cluster-group.aspx

In all instructions below the assumption is that the cluster service and appropriate quorum type have been established and fully configured.  The instructions also outline only the establishment of the Exchange resources, the TechNet documentation referenced above should be used to establish the specific cluster service requirements and other installation / management tasks for an Exchange 2007 SP1 clustered installation on Windows 2003.

Windows 2003 / Exchange 2007 SP1 Cluster Continuous Replication (CCR)

• On each node, run setup.com /mode:install /roles:mailbox.
  • This will establish the “passive” installation.
  • A reboot will be required post installation.
• Install the appropriate Exchange 2007 SP1 rollup update for your organization.
• On the “active” node, run setup.com /newCMS /cmsName:<NAME> /cmsIPAddress:<IPAddress>
  • This will establish the “active” installation.

At this point the replication service will begin copying log files from the new databases created and bring both sides into synchronization.  This completes the setup of a Windows 2003 / Exchange 2007 SP1 CCR cluster.

Windows 2003 / Exchange 2007 SP1 Single Copy Cluster (SCC)

• After establishing the clustered services any shared storage must be prepared for the Exchange installation.  Consider the following regarding shared storage:
  • Any lettered physical volume found during cluster creation will have a physical disk resource created for it and each of these resources placed into temporary groups.
  • When using cluster administrator, these groups will look like Group0 / Group1 / Group2 / etc each holding a single physical disk resource.
  • Any mountpoints established on the system are not automatically assigned physical disk resources in the cluster.
• To prepare the shared storage:
  • Create a new group within cluster administrator, name the group with a temporary name.  (Do not use the Exchange CMS name for this group).
  • Move all physical disk resources from their temporary group (GroupX) to the new group created.
  • If mountpoints are in use, create new physical disk resources in the temporary group you created for each mountpoint that exists on the node.
  • Mountpoint physical disk resources should be configured with a dependency of the lettered physical disk resources hosting the mountpoint.
  • Delete the temporary groups (GroupX) that should now be empty.
• On each node, run setup.com /mode:install /roles:mailbox
  • This will establish the “passive” installation.
  • A reboot will be required post installation.
• Install the appropriate Exchange 2007 SP1 rollup update for your organization.
• Using cluster administrator, move the group holding the physical disk resources to the “active” node.
• On the “active” node, run setup.com /newCMS /cmsName:<NAME> /cmsIPAddress:<IPAddress> /css /cmsDataPath:<Path to folder on shared disk>
  • This will establish the “active” installation storing the initial database and log files on the shared disk path specified in the command.
• Address the remaining shared storage requirements:
  • Physical disk resources must be moved from the temporary group you created to the Exchange CMS group created during setup.
  • As outlined in TechNet documentation, the dependencies for the database instances must be configured so that each database is dependant on the physical disks where the files reside.

By using these steps you should hopefully be able to simply your Exchange 2007 SP1 clustered installation on Windows 2003 and more easily address errors that may arise during the clustered setup.

The official TechNet documentation for establishing a Windows 2008 / Exchange 2007 SP1 cluster can be found at: 

• Exchange 2007 SP1 Single Copy Clusters (SCC) – http://technet.microsoft.com/en-us/library/bb125217.aspx
• Exchange 2007 SP1 Cluster Continuous Replication (CCR) – http://technet.microsoft.com/en-us/library/bb124521.aspx
• Installing Exchange 2007 SP1 Single Copy Clusters (SCC) on Windows 2008 – http://technet.microsoft.com/en-us/library/bb629581.aspx
• Installing Exchange 2007 SP1 Cluster Continuous Replication (CCR) on Windows 2008 – http://technet.microsoft.com/en-us/library/bb629714.aspx

What I want to do in this blog post is outline an alternate method for installing the cluster different from the instructions referenced above for Windows 2003 clustered nodes.

The first recommendation that I make when establishing the cluster service is to not use the GUI mode setup included with Exchange 2007.  My main rational for this advice is that when GUI mode setup fails you generally have to revert to command line setup to correct conditions and repeat the setup operations.  With this in mind it’s much easier to start with the command line setup and continue through.

In each of the links above you will see that the basic structure of a command line setup for the active roles installation is:

setup.com /mode:install /roles:mailbox /newCMS /cmsName /cmsIPAddress [/css] [/cmsDataPath]

The passive role installation is:

setup.com /mode:install /roles:mailbox

The way to think about the active role command is the combination of the passive role (setup.com /mode:install /roles:mailbox) with the establishment of the clustered services (setup.com /newCMS /cmsName /cmsIPAddress [/css] [/cmsDataPath]).

These commands do not have to be run together.  With this in mind my second recommendation is to establish the passive role first on each clustered node.  I recommend this for a few reasons:

• By separating the commands we can identify failures and address them easier.
• This method uses a foundational approach whereby the nodes are established and the cluster services verified, the mailbox roles established and verified, and only after that the clustered instance established and verified.
• When the commands are combined, and the failure occurs in the /newCMS portion of the command, further setup attempts using the same command error indicating the mailbox roles is currently installed and must be uninstalled.  Although the error is true, uninstalling the mailbox role and rerunning the same command without correcting the conditions that failed the cluster portion of setup will generally result is hitting the same error again (and thus the loop continues).
• In a shared storage environment the restart of the clustered services during mailbox role installation causes storage to move between nodes.  Setup attempts to move this storage back, but is sometimes not successful in doing so.  This causes the /newCMS portion of the setup command to fail, and thus the active role command to fail.  Separating the commands allows us to establish storage and arbitrate it to the “active” node for setup operations after the mailbox role has been installed.
• A setup code error exists where when an existing CMS exists in a cluster, and the passive node installation is run, the passive nodes machine account is promoted to full control permissions of the CMS.  This results in errors in the management console and management shell that an account was found with full administrator rights that is not an Exchange administrator.  (Note:  The error is truly benign, I just point it out because it is annoying.)

Now I will outline for you the steps that I use when establishing a Windows 2008 / Exchange 2007 SP1 Cluster.  Please refer to these additional blog posts for some other helpful information regarding these installations:

• http://blogs.technet.com/timmcmic/archive/2009/04/12/exchange-2007-renaming-the-default-storage-group-mailbox-database-or-deleting-and-recreating.aspx
• http://blogs.technet.com/timmcmic/archive/2009/04/15/one-thing-not-to-do-when-exchange-2007-cluster-setup-fails.aspx
• http://blogs.technet.com/timmcmic/archive/2009/04/26/file-share-witness-fsw-placement-and-the-cluster-group.aspx
• http://blogs.technet.com/timmcmic/archive/2009/04/26/windows-2008-multi-subnet-clusters-and-using-static-routes.aspx
• http://blogs.technet.com/timmcmic/archive/2009/02/24/errors-with-test-replicationhealth-when-using-a-multi-subnet-windows-2008-cluster.aspx
• http://blogs.technet.com/timmcmic/archive/2009/02/24/permissions-required-for-the-cno-cluster-name-object-in-windows-2008-for-exchange-2007-sp1-setup-operations.aspx
• http://blogs.technet.com/timmcmic/archive/2009/01/21/the-windows-cluster-service-encountered-and-error-during-function-opencluster.aspx
• http://blogs.technet.com/timmcmic/archive/2009/01/22/exchange-2007-sp1-ccr-windows-2008-clusters-file-share-witness-fsw-failures.aspx
• http://blogs.technet.com/timmcmic/archive/2008/12/23/exchange-replication-service-exchange-2007-sp1-and-windows-2008-clusters.aspx

In all instructions below the assumption is that the cluster service and appropriate quorum type have been established and fully configured.  The instructions also outline only the establishment of the Exchange resources, the TechNet documentation referenced above should be used to establish the specific cluster service requirements and other installation / management tasks for an Exchange 2007 SP1 clustered installation on Windows 2008.

Windows 2008 / Exchange 2007 SP1 Cluster Continuous Replication (CCR)

• On each node, run setup.com /mode:install /roles:mailbox.
  • This will establish the “passive” installation.
  • A reboot will be required post installation.
  • The installation command when run on a machine where the clustered service is configured and running simply installs the Exchange binaries to the local machine.
• Install the appropriate Exchange 2007 SP1 rollup update for your organization.
• On the “active” node, run setup.com /newCMS /cmsName:<NAME> /cmsIPv4Addresses:<IPAddress>,<IPAddress>
  • This will establish the “active” installation.

At this point the replication service will begin copying log files from the new databases created and bring both sides into synchronization.  This completes the setup of a Windows 2008 / Exchange 2007 SP1 CCR cluster.

Windows 2008 / Exchange 2007 SP1 Single Copy Cluster (SCC)

• After establishing the clustered services any shared storage must be prepared for the Exchange installation.  Consider the following regarding shared storage:
  • In Windows 2008 shared storage found at the time of cluster configuration is now placed into the “Available Storage” group.
  • The “Available Storage” group is actually a hidden group inside of cluster.  (You can see this group by running cluster <clusterFQDN> group from a command line.)
  • Each lettered physical drive and mountpoint found during cluster configuration will be represented in the available storage group by a physical disk resource.
  • Mountpoint physical disk resources are not automatically made dependant on the lettered physical disk resource hosting them.
• On each node, run setup.com /mode:install /roles:mailbox
  • This will establish the “passive” installation.
  • A reboot will be required post installation.
• Install the appropriate Exchange 2007 SP1 rollup update for your organization.
• Using the command line, move the available storage group to the node where you will run the “active” installation:
  • Open a command prompt.
  • Run cluster.exe <clusterFQDN> group “Available Storage” /moveto:<NODE>
  • For example, cluster.exe cluster-1.exchange.msft group “Available Storage” /moveto:node1.exchange.msft
• On the “active” node, run:
  • setup.com /newCMS /cmsName:<NAME> /cmsIPv4Addresses:<IPAddress>,<IPAddress> /css /cmsDataPath:<Path to folder on shared disk>
  • This will establish the “active” installation storing the initial database and log files on the shared disk path specified in the command.
• Address the remaining shared storage requirements:
  • Physical disk resources must be moved from the available storage group to the Exchange CMS group created during setup.
  • This can be accomplished by right clicking on the CMS group in cluster administrator and selecting add storage.
  • After the storage is moved, all mountpoint resources should be configured to have a dependency on the lettered physical disk hosting the mountpoint.
  • As outlined in TechNet documentation, the dependencies for the database instances must be configured so that each database is dependant on the physical disks where the files reside.

By using these steps you should hopefully be able to simply your Exchange 2007 SP1 clustered installation on Windows 2008 and more easily address errors that may arise during the clustered setup.