In Windows 2008 clusters, by default, all network name resources are enabled for Kerberos.  This causes the cluster service to create a machine account for the network name resource.  This is known a VCO or Virtual Computer Object.

When the machine account associated with a network name is deleted the network name in cluster will fail to come online.

There are events in the system log associated with this action which help to explain why.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          8/16/2009 3:31:40 PM
Event ID:      1207
Task Category: Network Name Resource
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      Node-2.domain.com

Description:
Cluster network name resource ‘Network Name (MBX-1)’ cannot be brought online. The computer object associated with the resource could not be updated in domain ‘domain.com’ for the following reason:
Unable to find computer account on DC where it was created.

The text for the associated error code is: There is no such object on the server.

The cluster identity ‘CLUSTER-1$’ may lack permissions required to update the object. Please work with your domain administrator to ensure that the cluster identity can update computer objects in the domain.

Cluster is aware of the DC where the object was created, and stamps this property as a private property of the network name resource.

cluster.exe <clusterFQDN> res “Network Name (MBX-1)” /priv

Listing private properties for ‘Network Name (MBX-1)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

BR Network Name (MBX-1) ResourceData                   01 00 00 00 … (260 bytes)

DR Network Name (MBX-1) StatusNetBIOS                  0 (0x0)

DR Network Name (MBX-1) StatusDNS                      0 (0x0)

DR Network Name (MBX-1) StatusKerberos                 8240 (0x2030)

SR Network Name (MBX-1) CreatingDC       \DC-1.domain.com

FTR Network Name (MBX-1) LastDNSUpdateTime              8/14/2009 3:07:59 AM

SR Network Name (MBX-1) ObjectGUID                     01e46402b3cc8a4fa124bd76a3801f69

S  Network Name (MBX-1) Name                           MBX-1

S  Network Name (MBX-1) DnsName                        MBX-1

D  Network Name (MBX-1) RemapPipeNames                 0 (0x0)

D  Network Name (MBX-1) RequireDNS                     0 (0x0)

D  Network Name (MBX-1) RequireKerberos                1 (0x1)

D  Network Name (MBX-1) HostRecordTTL                  1200 (0x4b0)

D  Network Name (MBX-1) RegisterAllProvidersIP         0 (0x0)

D  Network Name (MBX-1) PublishPTRRecords              0 (0x0)

D  Network Name (MBX-1) TimerCallbackAdditionalThreshold 5 (0x5)

D  Network Name (MBX-1) MSExchange_NetName             1 (0x1)

S  Network Name (MBX-1) RequireKerbero                 0

S  Network Name (MBX-1) requirekerbeoros               1

S  Network Name (MBX-1) requirekeberos   1

You’ll also note that the requireKerberos setting is set to 1 = enabled.

There are other ways to recover the VCO, but from an Exchange standpoint I find these to be the easiest…

1)  Create a new machine account in the desired container with the same name as the VCO / CNO.

2)  Using this blog post establish the permissions for the CNO on the new VCO.  (http://blogs.technet.com/timmcmic/archive/2009/02/24/permissions-required-for-the-cno-cluster-name-object-in-windows-2008-for-exchange-2007-sp1-setup-operations.aspx)

3)  Ensure the new machine account is disabled and allow time for ad replication.

4)  Ensure that you have your Exchange 2007 SP1 media on hand.

5)  Ensure that all resources in the CMS cluster group have been taken offline.

6)  Using the media and a command prompt, run the following command –> setup.com /clearLocalCMS.

7)  Recover the CMS to the cluster –> setup.com /recoverCMS /cmsName:<NAME> /cmsIPAddress:<IPAddress> or setup.com /recoverCMS /cmsName:<NAME> /cmsIPv4Addresses:<IPAddress1>,<IPAddress2>

When these steps are completed all Exchange resources should now be available and online as well as the new machine account created in an enabled state.

When Exchange 2007 is installed on a cluster there are several Exchange 2007 specific resources that are created.  These include:

• A system attendant resource.
• An information store resource.
• A database instance resource coordinating to each database created on the CMS.

Sometimes these clustered resources are accidentally deleted using cluster administrator or failover cluster manager.  This results in a portion of the solution not functioning.

Some clustered applications allow you to recreate individual clustered resources by using cluster administrator (or failover cluster management). 

Although the resource is created successfully within cluster, it will ultimately fail for Exchange use.

Each Exchange resource that is created by the integrated setup routine is stamped with Exchange specific values.  This is what allows the integration between Exchange and Windows cluster to function. 

Let’s take a look at some of these values.

Exchange System Attendant Instance (CMSName)

Listing private properties for ‘Exchange System Attendant Instance (2008-MBX3)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  Exchange System      NetworkName                    2008-MBX3

   Attendant Instance (2008-MBX3)         

The network name private property links the system attendant resource to the appropriate network name.  This value is not stamped by simply recreating the resource.

Exchange Information Store Instance (CMSName)

Listing private properties for ‘Exchange Information Store Instance (2008-MBX3)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

D  Exchange Information Store Instance (2008-MBX3) ResourceVersion                524289 (0x80001)

D  Exchange Information Store Instance (2008-MBX3) ResourceBuild                  671088646 (0x28000006)

S  Exchange Information Store Instance (2008-MBX3) NetworkName                    2008-MBX3

S  Exchange Information Store Instance (2008-MBX3) DestPath                       C:Program FilesMicrosoftExchange ServerMailboxMDBDATA

D  Exchange Information Store Instance (2008-MBX3) ClusteredStorageType           1 (0x1)

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplicaInitializing False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBroken False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyNotificationGenerationNumber 123

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyGenerationNumber 123

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_InspectorGenerationNumber 122

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplayGenerationNumber 121

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyNotificationTime 128853118429157196

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyTime 128853118429157196

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestInspectorTime 128854139574491853

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestReplayTime 128853118426031936

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CurrentReplayTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestFullBackupTime 128666480930000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestIncrementalBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestDifferentialBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestIncrementalBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestDifferentialBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestCopyBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LastFailoverTime 128661508136602054

S  Exchange Information Store Instance (2008-MBX3) LatestOnlineTime               128882659585976345

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyGenerationNumber 66

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyNotificationGenerationNumber 66

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_InspectorGenerationNumber 65

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplayGenerationNumber 64

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CurrentReplayTime 128882708597977356

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyNotificationTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestInspectorTime 128891788251028675

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestReplayTime 128882708591257832

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SuspendSuspendWanted False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SuspendMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LastFailoverTime 128661508146915082

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBroken False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBrokenMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ReplicaInitializing False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_CopyNotificationGenerationNumber 119

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_CopyGenerationNumber 119

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_InspectorGenerationNumber 118

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ReplayGenerationNumber 117

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyNotificationTime 128853118430094774

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyTime 128853118430094774

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestInspectorTime 128854139607929353

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestReplayTime 128853118426500725

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_CurrentReplayTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestFullBackupTime 128666481160000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestIncrementalBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestDifferentialBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestIncrementalBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestDifferentialBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestCopyBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_CopyGenerationNumber 64

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_CopyNotificationGenerationNumber 64

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_InspectorGenerationNumber 63

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_ReplayGenerationNumber 62

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_CurrentReplayTime 128882708597821088

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyNotificationTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestInspectorTime 128891788251028675

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestReplayTime 128882708593914388

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_SuspendSuspendWanted False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_SuspendMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LastFailoverTime 128661383610946829

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node2_58da7215-7d0c-4d18-835a-848bde0ce408_LastFailoverTime 128661383799855560

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryCreationTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryEndTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryRequired False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryStartTime 633572168518476877

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryCreationTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryEndTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryRequired False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryStartTime 633572168264869789

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryServers

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryServers

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBrokenMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestFullBackupTime 128860168890000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestFullBackupTime 128860169140000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_25b8ef30-3bae-474f-b075-8068fb524308_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestInspectorTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestInspectorTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestInspectorTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestInspectorTime 0

In this case the information store resource has several private properties that are not re-created by simply creating the resource.  These include the network name (similar to system attendant), the cluster storage type indicating the type of cluster used (CCR or SCC), and other private properties coordinating to the replication status of databases hosted on the server. 

Exchange Database Instances

Listing private properties for ‘2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) DatabaseGuid                   5be19b1d-845b-4a23-8aa2-d98abbd06274

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) StorageGroupGuid               58da7215-7d0c-4d18-835a-848bde0ce408

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) NetworkName                    2008-MBX3

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) LatestOfflineTime              128882708598133624

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) LastMountedOnServer            2008-NODE1

The database instances also have links that are missing when resources are created through cluster administrator.  For example, database instances are linked to their storage groups and databases by stamping GUIDs onto the cluster resource.  In this case the database guid is stamped into the private property DatabaseGuid and the storage group guid stamped into the private property StorageGroupGuid.  Without these attributes the database instances will not function.

It is possible in some instances to manually go back and re-stamp these private properties.  Particular care has to be taken to ensure that this happens correctly.  If it does not happen correctly unknown results may occur and the resource may not function.

IN GENERAL I DISCOURAGE ATTEMPTING TO MANUALLY RECREATE CLUSTERED RESOURCES!

In the event a deletion occurs, the following steps can be used to recover from the deletion.

1)  Navigate to the node that currently owns the Exchange resources.

2)  Make note of the CMS name and CMS IP address (properties of the Exchange network name resource and the Exchange IP resource).

3)  Using the Exchange Management Shell, issue a stop-clusteredmailboxserver.  Fill in the prompted information as necessary.  This will take the CMS offline.

4)  Using your Exchange 2007 SP1 media, issue a setup.com /clearLocalCMS /cmsName:<CMSName>.

http://technet.microsoft.com/en-us/library/cc164362.aspx

By clearing the CMS you have removed the clustered configuration associated with that CMS.

5)  Recover the CMS to the cluster by using the Exchange 2007 SP1 media and issuing setup.com /recoverCMS /cmsName:<CMSName> /cmsIPv4Addresses:<IPAddress>,<IPAddress> or setup.com /recoverCMS /cmsName:<CMSName> /cmsIPv4Address:<IPAddress>

http://technet.microsoft.com/en-us/library/bb124095.aspx

By recovering the CMS all clustered resources are refreshed and recreated.  This ensures that all attributes are stamped onto the cluster resources and the cluster should function as expected.

===================================================================================================

Update:  6/5/2011

===================================================================================================

I was recently contacted by a co-worker who found a more efficient way to recovery a deleted database instance from a clustered mailbox server.  Let’s take a look at an example:

In the example cluster MBX-3 I have two existing database instances:

Using Failover Cluster Manager I delete one of the database instances.  Here is the resulting view in Failover Cluster Manager:

Using the above instructions would require the administrator to recover the entire CMS.  It was found that if you create another storage group and database the database instances within the cluster are refreshed.  For example:

new-storagegroup –name <SG-NAME> –server <CMSName>

new-mailboxdatabase –name <DB-NAME> –storagegroup <CMSNAMESG-NAME>

This creates a new database instance within cluster for the new database and re-creates the missing database instance for you. 

The administrator can then remove the new mailbox database and storage group created and bring online the previously missing database instance.

remove-mailboxdatabase <DB-NAME>

remove-storagegroup <SG-NAME>

start-clusteredmailboxserver <CMSName>

These steps were verified on a Windows 2008 R2 / Exchange 2007 SP3 CCR cluster.  A special thanks to my co-worker Michael Barta for pointing this out to me!

===================================================================================================

When running Exchange 2007 SP1 and the command

setup.com /recoverCMS /cmsName:<NAME> /cmsIPAddress:<IP>

you may receive the following warning:

The installed version of Exchange Server 2007 may be different from the version you are trying to install. The current installed version is ‘8.1.359.2’, the last installed version was ‘8.1.240.6’.

In this case the version 8.1.240.6 represents Exchange 2007 SP1.  The version 8.1.359.2 represents Exchange 2007 SP1 RU7. 

One of the pre-req checks for recoverCMS is to warn when the version of Exchange installed on the node where recovery is run does not match the version recorded on the Exchange active directory object. 

The issue here is that when a rollup update is installed the version attribute in active directory is not updated (and this is by design).  For example, here is an LDP dump of an Exchange CMS object and it’s associated version.  Reviewing the serial number attribute you will see Version 8.1 (Build 30240.6).  This translates to Version 8.1.240.6 (combine the version and drop the 30).

Expanding base ‘CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft’…
Result <0>: (null)
Matched DNs:
Getting 1 entries:
>> Dn: CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
    3> objectClass: top; server; msExchExchangeServer;
    1> cn: 2003-MBX1;
   1> serialNumber: Version 8.1 (Build 30240.6);
    1> distinguishedName: CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
    1> instanceType: 0x4 = ( IT_WRITE );
    1> whenCreated: 10/01/2008 07:07:30 Eastern Standard Time Eastern Standard Time;
    1> whenChanged: 10/01/2008 07:19:24 Eastern Standard Time Eastern Standard Time;
    1> uSNCreated: 37192;
    1> uSNChanged: 37276;
    1> showInAdvancedViewOnly: TRUE;
    1> adminDisplayName: 2003-MBX1;
    6> networkAddress: ncacn_vns_spp:2003-MBX1; netbios:2003-MBX1; ncacn_np:2003-MBX1; ncacn_spx:2003-MBX1; ncacn_ip_tcp:2003-MBX1.exchange.msft; ncalrpc:2003-MBX1;
    1> name: 2003-MBX1;
    1> objectGUID: d7452d19-806a-43ea-b163-24d265f096d7;
    1> versionNumber: 1912701168;
    1> serverRole: 0;
    1> systemFlags: 0x52000000 = ( FLAG_CONFIG_ALLOW_RENAME | FLAG_CONFIG_ALLOW_LIMITED_MOVE | FLAG_DISALLOW_MOVE_ON_DELETE );
    1> legacyExchangeDN: /o=Microsoft/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=2003-MBX1;
    1> objectCategory: CN=ms-Exch-Exchange-Server,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
    1> msExchTransportTransientFailureRetryInterval: 300;
    1> msExchTransportMessageRetryInterval: 60;
    1> msExchTransportInternalMaxDSNMessageAttachmentSize: 10485760;
    1> msExchMailboxManagerActivationSchedule: ;
    1> msExchTransportMessageTrackingPath: C:Program FilesMicrosoftExchange ServerTransportRolesLogsMessageTracking;
    1> msExchTransportMaxConcurrentMailboxSubmissions: 20;
    1> msExchMailboxManagerActivationStyle: 0;
    1> type: <ldp: Binary blob>;
    1> msExchMessageTrackLogFilter: -262145;
    1> msExchMinAdminVersion: -2147453113;
    1> msExchInstallPath: C:Program FilesMicrosoftExchange Server;
    1> msExchMailboxManagerAdminMode: 2;
    1> msExchTransportMaxConnectivityLogAge: 2592000;
    1> msExchTransportOutboundConnectionFailureRetryInterval: 600;
    1> msExchTransportMaxPickupDirectoryMessagesPerMinute: 100;
    1> messageTrackingEnabled: FALSE;
    1> msExchServerSite: CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=exchange,DC=msft;
    1> msExchTransportMaxConcurrentMailboxDeliveries: 7;
    1> msExchServerRole: 0;
    1> msExchEdgeSyncAdamSSLPort: 50636;
    1> msExchTransportMaxMessageTrackingDirectorySize: 262144000;
    1> msExchTrkLogCleaningInterval: 7;
    1> msExchTransportDelayNotificationTimeout: 14400;
    1> msExchTransportRoutingLogMaxAge: 604800;
    1> msExchTransportMaxMessageTrackingFileSize: 10485760;
    1> msExchTransportExternalDefaultLanguage: en-US;
    1> msExchResponsibleMTAServer: CN=Microsoft MTA,CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
    1> msExchCurrentServerRoles: 2;
    1> msExchTransportMaxMessageTrackingLogAge: 2592000;
    1> msExchTransportPoisonMessageThreshold: 2;
    1> msExchDataLossForAutoDatabaseMount: 0;
    1> msExchTransportMaxPickupDirectoryHeaderSize: 65536;
    1> msExchTransportMaxPickupDirectoryRecipients: 100;
    1> msExchDataPath: C:Program FilesMicrosoftExchange ServerMailbox;
    1> msExchSmtpReceiveMaxConnectionRatePerMinute: 1200;
    1> msExchTransportMaxQueueIdleTime: 180;
    1> msExchTransportMaxReceiveProtocolLogAge: 2592000;
    1> msExchTransportExternalMaxDSNMessageAttachmentSize: 10485760;
    1> heuristics: 268435456;
    1> msExchTransportFlags: 17401;
    1> msExchMonitoringResources: 2:1:Default Microsoft Exchange Services:MSExchangeSA:MSExchangeMTA:RESvc:SMTPSVC:MSExchangeIS:W3SVC:;
    1> msExchTransportInternalDefaultLanguage: en-US;
    1> msExchELCAuditLogFileAgeLimit: 0;
    1> msExchHomeRoutingGroup: CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
    1> msExchTransportMaxSendProtocolLogAge: 2592000;
    1> msExchELCAuditLogFileSizeLimit: 10485760;
    1> msExchELCAuditLogPath: C:Program FilesMicrosoftExchange ServerLoggingManaged Folder Assistant;
    1> msExchTransportTransientFailureRetryCount: 6;
    1> msExchTransportMessageExpirationTimeout: 172800;
    1> msExchVersion: 4535486012416;
———–

Now we know how to determine the Exchange version based on what is stamped in active directory, how do we determine the version of Exchange installed on the machine.  The version of Exchange installed on the machine is determined by the version of ExSETUP.exe installed on the machine.  ExSETUP.exe can be found in X:Program FilesMicrosoftExchange ServerBIN (assuming X is the drive where Exchange was installed and that the default path was used).  Here is an example version of ExSETUP.exe.

ExSETUP is updated with each rollup that is installed onto the machine. 

Version 8.1.240.6 != 8.1.359.2 or another way of saying it Version 8.1.240.6 ≠ 8.1.359.2 and the error / warning is thrown.

With later version of the XML pre-reqs file this text is a warning.  With prior version of the XML pre-reqs file this is a hard stop error preventing /recoverCMS from completing successfully. 

If you have hit the error condition, and cannot proceed, the following instructions should allow you to continue:

1)  Find and locate ExSETUP.exe in X:Program FilesMicrosoftExchange ServerBin.

2)  Rename the local version of ExSETUP.exe to ExSETUP.exe.original.

3)  From your Exchange installation media, locate the version of ExSETUP.exe.  (MediaSetupServerRolesCommon).

4)  Copy this ExSETUP.exe to the bin directory.

5)  Run your setup command and allow it to complete successfully.

6)  When completed, either reinstall the last rollup update applied or delete the copied ExSETUP.exe and rename the original back to ExSETUP.exe.

When planning an Exchange 2007 installation storage considerations are a very important factor.  I wanted to call attention to a sizing consideration that may not be accounted for in other places.  This sizing consideration should be accounted for when planning an environment that may use Cluster Continuous Replication, Local Continuous Replication, or Standby Continuous Replication.

When planning your log file drive sizing it is important to consider when log files are actually purged.  It is important to remember that log file purging cannot occur unless:

• CCR – The log has been copied, inspected, and replayed.
• LCR – The log has been copied, inspected, and replayed.
• SCR – The log has been copied, inspected, and put out for replay at a later time.

When the above criteria cannot be met log files will continue to fill the log file volume even if a backup is performed successfully.  If the criteria above are not met, and a backup is performed successfully, logs will not be truncated.

Depending on the number of days that the replication partner is not available, this may result in a large number of log files remaining on the log file drive and in some instances a log file drive full condition results.  When the log file drive is full and logs can no longer be created, the database instance(s) will dismount and be unavailable.

This would require the administrator to either return the log copy target to availability so that logs can be copied and purged or to purge log files manually.  In the event log files are purged manually a full database re-seed of the passive target would be necessary.

In planning you should consider factors that might cause nodes to be unavailable for an extended period of time – for example WAN issues.  If necessary increase the size of your log file volume to accommodate for periods where replication cannot occur.  For example, if your log generation per day is estimated at 2000 logs, and you estimate that any outage of a node or network etc could last up to 5 days, you need to make plans to accommodate up to 5 days of non-replicated log files.

The storage calculator can assist you in your planning.  There are two areas of the storage calculator that help you take into account planned outages of both replication and backups in order to provide a more accurate log file drive sizing.  The two areas of the storage calculator that can help you better estimate log file volume size for backup and network issues are:

• In Step 3 – Backup Configuration – Backup Failure Tolerance
  • The backup failure tolerance allows you to choose how many days you can go without a backup that performs truncation.  Full Backups and Incremental backups purge the transaction log files since the last full / incremental backup.  However if a backup job fails you need to ensure that you have enough capacity to allow for either restoration or continuation of service until the next backup window.

• In Step 4 – Storage Requirements Input Factors – Log Replication Configuration – Network Fault Tolerance (Days)
  • When deploying geographically dispersed CCR and SCR across a WAN link there is the possibility that the network link between the two locations will become unavailable.  As a result, truncation on the source cannot occur.  To ensure you have enough space to survive the network outage, enter a value for Network Fault Tolerance (measured in days).

Download the storage calculator here –

http://msexchangeteam.com/files/12/attachments/entry438481.aspx

Read about the storage calculator here –

http://msexchangeteam.com/archive/2007/01/15/432207.aspx

By planning for lack of replication ahead of time you can hopefully ward off any out of disk space conditions or conditions that may cause a full re-seed to become necessary.