In Windows 2008 clusters, by default, all network name resources are enabled for Kerberos.  This causes the cluster service to create a machine account for the network name resource.  This is known a VCO or Virtual Computer Object.

When the machine account associated with a network name is deleted the network name in cluster will fail to come online.

There are events in the system log associated with this action which help to explain why.

Log Name:      System
Source:        Microsoft-Windows-FailoverClustering
Date:          8/16/2009 3:31:40 PM
Event ID:      1207
Task Category: Network Name Resource
Level:         Error
Keywords:     
User:          SYSTEM
Computer:      Node-2.domain.com

Description:
Cluster network name resource ‘Network Name (MBX-1)’ cannot be brought online. The computer object associated with the resource could not be updated in domain ‘domain.com’ for the following reason:
Unable to find computer account on DC where it was created.

The text for the associated error code is: There is no such object on the server.

The cluster identity ‘CLUSTER-1$’ may lack permissions required to update the object. Please work with your domain administrator to ensure that the cluster identity can update computer objects in the domain.

Cluster is aware of the DC where the object was created, and stamps this property as a private property of the network name resource.

cluster.exe <clusterFQDN> res “Network Name (MBX-1)” /priv

Listing private properties for ‘Network Name (MBX-1)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

BR Network Name (MBX-1) ResourceData                   01 00 00 00 … (260 bytes)

DR Network Name (MBX-1) StatusNetBIOS                  0 (0x0)

DR Network Name (MBX-1) StatusDNS                      0 (0x0)

DR Network Name (MBX-1) StatusKerberos                 8240 (0x2030)

SR Network Name (MBX-1) CreatingDC       \DC-1.domain.com

FTR Network Name (MBX-1) LastDNSUpdateTime              8/14/2009 3:07:59 AM

SR Network Name (MBX-1) ObjectGUID                     01e46402b3cc8a4fa124bd76a3801f69

S  Network Name (MBX-1) Name                           MBX-1

S  Network Name (MBX-1) DnsName                        MBX-1

D  Network Name (MBX-1) RemapPipeNames                 0 (0x0)

D  Network Name (MBX-1) RequireDNS                     0 (0x0)

D  Network Name (MBX-1) RequireKerberos                1 (0x1)

D  Network Name (MBX-1) HostRecordTTL                  1200 (0x4b0)

D  Network Name (MBX-1) RegisterAllProvidersIP         0 (0x0)

D  Network Name (MBX-1) PublishPTRRecords              0 (0x0)

D  Network Name (MBX-1) TimerCallbackAdditionalThreshold 5 (0x5)

D  Network Name (MBX-1) MSExchange_NetName             1 (0x1)

S  Network Name (MBX-1) RequireKerbero                 0

S  Network Name (MBX-1) requirekerbeoros               1

S  Network Name (MBX-1) requirekeberos   1

You’ll also note that the requireKerberos setting is set to 1 = enabled.

There are other ways to recover the VCO, but from an Exchange standpoint I find these to be the easiest…

1)  Create a new machine account in the desired container with the same name as the VCO / CNO.

2)  Using this blog post establish the permissions for the CNO on the new VCO.  (http://blogs.technet.com/timmcmic/archive/2009/02/24/permissions-required-for-the-cno-cluster-name-object-in-windows-2008-for-exchange-2007-sp1-setup-operations.aspx)

3)  Ensure the new machine account is disabled and allow time for ad replication.

4)  Ensure that you have your Exchange 2007 SP1 media on hand.

5)  Ensure that all resources in the CMS cluster group have been taken offline.

6)  Using the media and a command prompt, run the following command –> setup.com /clearLocalCMS.

7)  Recover the CMS to the cluster –> setup.com /recoverCMS /cmsName:<NAME> /cmsIPAddress:<IPAddress> or setup.com /recoverCMS /cmsName:<NAME> /cmsIPv4Addresses:<IPAddress1>,<IPAddress2>

When these steps are completed all Exchange resources should now be available and online as well as the new machine account created in an enabled state.

When Exchange 2007 is installed on a cluster there are several Exchange 2007 specific resources that are created.  These include:

• A system attendant resource.
• An information store resource.
• A database instance resource coordinating to each database created on the CMS.

Sometimes these clustered resources are accidentally deleted using cluster administrator or failover cluster manager.  This results in a portion of the solution not functioning.

Some clustered applications allow you to recreate individual clustered resources by using cluster administrator (or failover cluster management). 

Although the resource is created successfully within cluster, it will ultimately fail for Exchange use.

Each Exchange resource that is created by the integrated setup routine is stamped with Exchange specific values.  This is what allows the integration between Exchange and Windows cluster to function. 

Let’s take a look at some of these values.

Exchange System Attendant Instance (CMSName)

Listing private properties for ‘Exchange System Attendant Instance (2008-MBX3)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  Exchange System      NetworkName                    2008-MBX3

   Attendant Instance (2008-MBX3)         

The network name private property links the system attendant resource to the appropriate network name.  This value is not stamped by simply recreating the resource.

Exchange Information Store Instance (CMSName)

Listing private properties for ‘Exchange Information Store Instance (2008-MBX3)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

D  Exchange Information Store Instance (2008-MBX3) ResourceVersion                524289 (0x80001)

D  Exchange Information Store Instance (2008-MBX3) ResourceBuild                  671088646 (0x28000006)

S  Exchange Information Store Instance (2008-MBX3) NetworkName                    2008-MBX3

S  Exchange Information Store Instance (2008-MBX3) DestPath                       C:Program FilesMicrosoftExchange ServerMailboxMDBDATA

D  Exchange Information Store Instance (2008-MBX3) ClusteredStorageType           1 (0x1)

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplicaInitializing False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBroken False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyNotificationGenerationNumber 123

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyGenerationNumber 123

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_InspectorGenerationNumber 122

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplayGenerationNumber 121

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyNotificationTime 128853118429157196

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyTime 128853118429157196

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestInspectorTime 128854139574491853

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestReplayTime 128853118426031936

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CurrentReplayTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestFullBackupTime 128666480930000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestIncrementalBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestDifferentialBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestIncrementalBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestDifferentialBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestCopyBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LastFailoverTime 128661508136602054

S  Exchange Information Store Instance (2008-MBX3) LatestOnlineTime               128882659585976345

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyGenerationNumber 66

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CopyNotificationGenerationNumber 66

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_InspectorGenerationNumber 65

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ReplayGenerationNumber 64

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_CurrentReplayTime 128882708597977356

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestCopyNotificationTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestInspectorTime 128891788251028675

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestReplayTime 128882708591257832

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SuspendSuspendWanted False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SuspendMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LastFailoverTime 128661508146915082

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBroken False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ConfigBrokenMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ReplicaInitializing False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_CopyNotificationGenerationNumber 119

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_CopyGenerationNumber 119

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_InspectorGenerationNumber 118

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_ReplayGenerationNumber 117

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyNotificationTime 128853118430094774

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyTime 128853118430094774

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestInspectorTime 128854139607929353

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestReplayTime 128853118426500725

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_CurrentReplayTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_NoLoss True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_MountAllowed True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestFullBackupTime 128666481160000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestIncrementalBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestDifferentialBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyBackupTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestIncrementalBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestDifferentialBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestCopyBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_CopyGenerationNumber 64

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_CopyNotificationGenerationNumber 64

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_InspectorGenerationNumber 63

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_ReplayGenerationNumber 62

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_CurrentReplayTime 128882708597821088

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestCopyNotificationTime 128882708598133624

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestInspectorTime 128891788251028675

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestReplayTime 128882708593914388

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_SuspendSuspendWanted False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_SuspendMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_Seeding False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LastFailoverTime 128661383610946829

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node2_58da7215-7d0c-4d18-835a-848bde0ce408_LastFailoverTime 128661383799855560

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryCreationTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryEndTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryRequired False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryStartTime 633572168518476877

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryCreationTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryEndTime 180000000000

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryRequired False

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryStartTime 633572168264869789

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408_DumpsterRedeliveryServers

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5_DumpsterRedeliveryServers

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-Node1_7096c806-d69d-41b8-ae1d-50ada0b0dce5_ConfigBrokenMessage

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_LatestFullBackupTime 128860168890000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_7096c806-d69d-41b8-ae1d-50ada0b0dce5_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_LatestFullBackupTime 128860169140000000

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_58da7215-7d0c-4d18-835a-848bde0ce408_SnapshotLatestFullBackup False

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-node2_25b8ef30-3bae-474f-b075-8068fb524308_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_SuspendCurrentOwner Idle

S  Exchange Information Store Instance (2008-MBX3) Replay_[LOCKS]_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ReplicaInitializing True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestInspectorTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE1.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestInspectorTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBroken True

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_ConfigBrokenMessage Status information cannot be displayed correctly because the storage group is running on a later version of Exchange Server than the client that is requesting the status information.

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_TargetReplicaInstanceState NotRunning

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_7096c806-d69d-41b8-ae1d-50ada0b0dce5|Standby_LatestInspectorTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestCopyNotificationTime 0

S  Exchange Information Store Instance (2008-MBX3) Replay_2008-NODE2.exchange.msft_58da7215-7d0c-4d18-835a-848bde0ce408|Standby_LatestInspectorTime 0

In this case the information store resource has several private properties that are not re-created by simply creating the resource.  These include the network name (similar to system attendant), the cluster storage type indicating the type of cluster used (CCR or SCC), and other private properties coordinating to the replication status of databases hosted on the server. 

Exchange Database Instances

Listing private properties for ‘2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3)’:

T  Resource             Name                           Value

— ——————– —————————— ———————–

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) DatabaseGuid                   5be19b1d-845b-4a23-8aa2-d98abbd06274

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) StorageGroupGuid               58da7215-7d0c-4d18-835a-848bde0ce408

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) NetworkName                    2008-MBX3

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) LatestOfflineTime              128882708598133624

S  2008-MBX3-SG2/2008-MBX3-SG2-DB1 (2008-MBX3) LastMountedOnServer            2008-NODE1

The database instances also have links that are missing when resources are created through cluster administrator.  For example, database instances are linked to their storage groups and databases by stamping GUIDs onto the cluster resource.  In this case the database guid is stamped into the private property DatabaseGuid and the storage group guid stamped into the private property StorageGroupGuid.  Without these attributes the database instances will not function.

It is possible in some instances to manually go back and re-stamp these private properties.  Particular care has to be taken to ensure that this happens correctly.  If it does not happen correctly unknown results may occur and the resource may not function.

IN GENERAL I DISCOURAGE ATTEMPTING TO MANUALLY RECREATE CLUSTERED RESOURCES!

In the event a deletion occurs, the following steps can be used to recover from the deletion.

1)  Navigate to the node that currently owns the Exchange resources.

2)  Make note of the CMS name and CMS IP address (properties of the Exchange network name resource and the Exchange IP resource).

3)  Using the Exchange Management Shell, issue a stop-clusteredmailboxserver.  Fill in the prompted information as necessary.  This will take the CMS offline.

4)  Using your Exchange 2007 SP1 media, issue a setup.com /clearLocalCMS /cmsName:<CMSName>.

http://technet.microsoft.com/en-us/library/cc164362.aspx

By clearing the CMS you have removed the clustered configuration associated with that CMS.

5)  Recover the CMS to the cluster by using the Exchange 2007 SP1 media and issuing setup.com /recoverCMS /cmsName:<CMSName> /cmsIPv4Addresses:<IPAddress>,<IPAddress> or setup.com /recoverCMS /cmsName:<CMSName> /cmsIPv4Address:<IPAddress>

http://technet.microsoft.com/en-us/library/bb124095.aspx

By recovering the CMS all clustered resources are refreshed and recreated.  This ensures that all attributes are stamped onto the cluster resources and the cluster should function as expected.

===================================================================================================

Update:  6/5/2011

===================================================================================================

I was recently contacted by a co-worker who found a more efficient way to recovery a deleted database instance from a clustered mailbox server.  Let’s take a look at an example:

In the example cluster MBX-3 I have two existing database instances:

Using Failover Cluster Manager I delete one of the database instances.  Here is the resulting view in Failover Cluster Manager:

Using the above instructions would require the administrator to recover the entire CMS.  It was found that if you create another storage group and database the database instances within the cluster are refreshed.  For example:

new-storagegroup –name <SG-NAME> –server <CMSName>

new-mailboxdatabase –name <DB-NAME> –storagegroup <CMSNAMESG-NAME>

This creates a new database instance within cluster for the new database and re-creates the missing database instance for you. 

The administrator can then remove the new mailbox database and storage group created and bring online the previously missing database instance.

remove-mailboxdatabase <DB-NAME>

remove-storagegroup <SG-NAME>

start-clusteredmailboxserver <CMSName>

These steps were verified on a Windows 2008 R2 / Exchange 2007 SP3 CCR cluster.  A special thanks to my co-worker Michael Barta for pointing this out to me!

===================================================================================================

When running Exchange 2007 SP1 and the command

setup.com /recoverCMS /cmsName:<NAME> /cmsIPAddress:<IP>

you may receive the following warning:

The installed version of Exchange Server 2007 may be different from the version you are trying to install. The current installed version is ‘8.1.359.2’, the last installed version was ‘8.1.240.6’.

In this case the version 8.1.240.6 represents Exchange 2007 SP1.  The version 8.1.359.2 represents Exchange 2007 SP1 RU7. 

One of the pre-req checks for recoverCMS is to warn when the version of Exchange installed on the node where recovery is run does not match the version recorded on the Exchange active directory object. 

The issue here is that when a rollup update is installed the version attribute in active directory is not updated (and this is by design).  For example, here is an LDP dump of an Exchange CMS object and it’s associated version.  Reviewing the serial number attribute you will see Version 8.1 (Build 30240.6).  This translates to Version 8.1.240.6 (combine the version and drop the 30).

Expanding base ‘CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft’…
Result <0>: (null)
Matched DNs:
Getting 1 entries:
>> Dn: CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
    3> objectClass: top; server; msExchExchangeServer;
    1> cn: 2003-MBX1;
   1> serialNumber: Version 8.1 (Build 30240.6);
    1> distinguishedName: CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
    1> instanceType: 0x4 = ( IT_WRITE );
    1> whenCreated: 10/01/2008 07:07:30 Eastern Standard Time Eastern Standard Time;
    1> whenChanged: 10/01/2008 07:19:24 Eastern Standard Time Eastern Standard Time;
    1> uSNCreated: 37192;
    1> uSNChanged: 37276;
    1> showInAdvancedViewOnly: TRUE;
    1> adminDisplayName: 2003-MBX1;
    6> networkAddress: ncacn_vns_spp:2003-MBX1; netbios:2003-MBX1; ncacn_np:2003-MBX1; ncacn_spx:2003-MBX1; ncacn_ip_tcp:2003-MBX1.exchange.msft; ncalrpc:2003-MBX1;
    1> name: 2003-MBX1;
    1> objectGUID: d7452d19-806a-43ea-b163-24d265f096d7;
    1> versionNumber: 1912701168;
    1> serverRole: 0;
    1> systemFlags: 0x52000000 = ( FLAG_CONFIG_ALLOW_RENAME | FLAG_CONFIG_ALLOW_LIMITED_MOVE | FLAG_DISALLOW_MOVE_ON_DELETE );
    1> legacyExchangeDN: /o=Microsoft/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=2003-MBX1;
    1> objectCategory: CN=ms-Exch-Exchange-Server,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
    1> msExchTransportTransientFailureRetryInterval: 300;
    1> msExchTransportMessageRetryInterval: 60;
    1> msExchTransportInternalMaxDSNMessageAttachmentSize: 10485760;
    1> msExchMailboxManagerActivationSchedule: ;
    1> msExchTransportMessageTrackingPath: C:Program FilesMicrosoftExchange ServerTransportRolesLogsMessageTracking;
    1> msExchTransportMaxConcurrentMailboxSubmissions: 20;
    1> msExchMailboxManagerActivationStyle: 0;
    1> type: <ldp: Binary blob>;
    1> msExchMessageTrackLogFilter: -262145;
    1> msExchMinAdminVersion: -2147453113;
    1> msExchInstallPath: C:Program FilesMicrosoftExchange Server;
    1> msExchMailboxManagerAdminMode: 2;
    1> msExchTransportMaxConnectivityLogAge: 2592000;
    1> msExchTransportOutboundConnectionFailureRetryInterval: 600;
    1> msExchTransportMaxPickupDirectoryMessagesPerMinute: 100;
    1> messageTrackingEnabled: FALSE;
    1> msExchServerSite: CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=exchange,DC=msft;
    1> msExchTransportMaxConcurrentMailboxDeliveries: 7;
    1> msExchServerRole: 0;
    1> msExchEdgeSyncAdamSSLPort: 50636;
    1> msExchTransportMaxMessageTrackingDirectorySize: 262144000;
    1> msExchTrkLogCleaningInterval: 7;
    1> msExchTransportDelayNotificationTimeout: 14400;
    1> msExchTransportRoutingLogMaxAge: 604800;
    1> msExchTransportMaxMessageTrackingFileSize: 10485760;
    1> msExchTransportExternalDefaultLanguage: en-US;
    1> msExchResponsibleMTAServer: CN=Microsoft MTA,CN=2003-MBX1,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
    1> msExchCurrentServerRoles: 2;
    1> msExchTransportMaxMessageTrackingLogAge: 2592000;
    1> msExchTransportPoisonMessageThreshold: 2;
    1> msExchDataLossForAutoDatabaseMount: 0;
    1> msExchTransportMaxPickupDirectoryHeaderSize: 65536;
    1> msExchTransportMaxPickupDirectoryRecipients: 100;
    1> msExchDataPath: C:Program FilesMicrosoftExchange ServerMailbox;
    1> msExchSmtpReceiveMaxConnectionRatePerMinute: 1200;
    1> msExchTransportMaxQueueIdleTime: 180;
    1> msExchTransportMaxReceiveProtocolLogAge: 2592000;
    1> msExchTransportExternalMaxDSNMessageAttachmentSize: 10485760;
    1> heuristics: 268435456;
    1> msExchTransportFlags: 17401;
    1> msExchMonitoringResources: 2:1:Default Microsoft Exchange Services:MSExchangeSA:MSExchangeMTA:RESvc:SMTPSVC:MSExchangeIS:W3SVC:;
    1> msExchTransportInternalDefaultLanguage: en-US;
    1> msExchELCAuditLogFileAgeLimit: 0;
    1> msExchHomeRoutingGroup: CN=Exchange Routing Group (DWBGZMFD01QNBJR),CN=Routing Groups,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=Microsoft,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
    1> msExchTransportMaxSendProtocolLogAge: 2592000;
    1> msExchELCAuditLogFileSizeLimit: 10485760;
    1> msExchELCAuditLogPath: C:Program FilesMicrosoftExchange ServerLoggingManaged Folder Assistant;
    1> msExchTransportTransientFailureRetryCount: 6;
    1> msExchTransportMessageExpirationTimeout: 172800;
    1> msExchVersion: 4535486012416;
———–

Now we know how to determine the Exchange version based on what is stamped in active directory, how do we determine the version of Exchange installed on the machine.  The version of Exchange installed on the machine is determined by the version of ExSETUP.exe installed on the machine.  ExSETUP.exe can be found in X:Program FilesMicrosoftExchange ServerBIN (assuming X is the drive where Exchange was installed and that the default path was used).  Here is an example version of ExSETUP.exe.

ExSETUP is updated with each rollup that is installed onto the machine. 

Version 8.1.240.6 != 8.1.359.2 or another way of saying it Version 8.1.240.6 ≠ 8.1.359.2 and the error / warning is thrown.

With later version of the XML pre-reqs file this text is a warning.  With prior version of the XML pre-reqs file this is a hard stop error preventing /recoverCMS from completing successfully. 

If you have hit the error condition, and cannot proceed, the following instructions should allow you to continue:

1)  Find and locate ExSETUP.exe in X:Program FilesMicrosoftExchange ServerBin.

2)  Rename the local version of ExSETUP.exe to ExSETUP.exe.original.

3)  From your Exchange installation media, locate the version of ExSETUP.exe.  (MediaSetupServerRolesCommon).

4)  Copy this ExSETUP.exe to the bin directory.

5)  Run your setup command and allow it to complete successfully.

6)  When completed, either reinstall the last rollup update applied or delete the copied ExSETUP.exe and rename the original back to ExSETUP.exe.

When planning an Exchange 2007 installation storage considerations are a very important factor.  I wanted to call attention to a sizing consideration that may not be accounted for in other places.  This sizing consideration should be accounted for when planning an environment that may use Cluster Continuous Replication, Local Continuous Replication, or Standby Continuous Replication.

When planning your log file drive sizing it is important to consider when log files are actually purged.  It is important to remember that log file purging cannot occur unless:

• CCR – The log has been copied, inspected, and replayed.
• LCR – The log has been copied, inspected, and replayed.
• SCR – The log has been copied, inspected, and put out for replay at a later time.

When the above criteria cannot be met log files will continue to fill the log file volume even if a backup is performed successfully.  If the criteria above are not met, and a backup is performed successfully, logs will not be truncated.

Depending on the number of days that the replication partner is not available, this may result in a large number of log files remaining on the log file drive and in some instances a log file drive full condition results.  When the log file drive is full and logs can no longer be created, the database instance(s) will dismount and be unavailable.

This would require the administrator to either return the log copy target to availability so that logs can be copied and purged or to purge log files manually.  In the event log files are purged manually a full database re-seed of the passive target would be necessary.

In planning you should consider factors that might cause nodes to be unavailable for an extended period of time – for example WAN issues.  If necessary increase the size of your log file volume to accommodate for periods where replication cannot occur.  For example, if your log generation per day is estimated at 2000 logs, and you estimate that any outage of a node or network etc could last up to 5 days, you need to make plans to accommodate up to 5 days of non-replicated log files.

The storage calculator can assist you in your planning.  There are two areas of the storage calculator that help you take into account planned outages of both replication and backups in order to provide a more accurate log file drive sizing.  The two areas of the storage calculator that can help you better estimate log file volume size for backup and network issues are:

• In Step 3 – Backup Configuration – Backup Failure Tolerance
  • The backup failure tolerance allows you to choose how many days you can go without a backup that performs truncation.  Full Backups and Incremental backups purge the transaction log files since the last full / incremental backup.  However if a backup job fails you need to ensure that you have enough capacity to allow for either restoration or continuation of service until the next backup window.

• In Step 4 – Storage Requirements Input Factors – Log Replication Configuration – Network Fault Tolerance (Days)
  • When deploying geographically dispersed CCR and SCR across a WAN link there is the possibility that the network link between the two locations will become unavailable.  As a result, truncation on the source cannot occur.  To ensure you have enough space to survive the network outage, enter a value for Network Fault Tolerance (measured in days).

Download the storage calculator here –

http://msexchangeteam.com/files/12/attachments/entry438481.aspx

Read about the storage calculator here –

http://msexchangeteam.com/archive/2007/01/15/432207.aspx

By planning for lack of replication ahead of time you can hopefully ward off any out of disk space conditions or conditions that may cause a full re-seed to become necessary.

Many customers have requested instructions on how to enable standby continuous replication to use an alternate network interface.  By design standby continuous replication always uses the “public” interface to ship logs and seed the database.

Over the past few weeks we have been working with the Exchange product group on a “supported” method to allow standby continuous replication to use an alternate network interface.  This blog will detail how to implement these steps and what effects it has on the overall solution.

First if you are reading this post you should review the replication service deep dive whitepaper located at http://technet.microsoft.com/en-us/library/cc535020.aspx (“White Paper:  Continuous Replication Deep Dive").  When reviewing this whitepaper it is important to pay attention to what sources are involved in replication when using standby continuous replication.  For example:

• When the source is a standalone mailbox server, logs are replicated by connecting to shares mapped to the server name.
• When the source is a cluster continuous replication server, logs are replicated by connecting to shares mapped to the ACTIVE node server name.
• When the source is a single copy cluster, logs are replicated by connecting to shares mapped to the clustered mailbox server name.

Keeping these parameters in mind will help you understand how the following changes will allow for standby continuous replication to use an alternate network interface.

The steps to implement this vary little by operating system.  Windows 2008 though does introduce some changes to the way file shares are handled.  Please review this blog for information on how share scoping in Windows 2008 effects the operations of the replication service.  (http://blogs.technet.com/timmcmic/archive/2008/12/23/exchange-replication-service-exchange-2007-sp1-and-windows-2008-clusters.aspx)

The following instructions are based on Exchange 2007 SP1 with RU7.  All customers implementing these instructions are encouraged to do so on Exchange 2007 SP1 RU7.

Replication behavior when using standby continuous replication over an alternate network interface.

When the instructions are implemented as documented, all network traffic from the SCR target to the SCR source is first routed through the private interface.  This can be verified with netmon by reviewing SMB (Windows 2003) or SMBv2 (Windows 2008) traffic. 

It is important to note that these instructions only effect the LOG SHIPPING functionality of SCR.  Other functions such as update-storagegroupcopy will only occur using the public interface.  This requires that both the source and target have the ability to communicate over both the public and private interfaces.  Planning for network sizing should take into account that re-seeding operations using update-storagegroupcopy must occur over the public interface.

Unlike continuous replication host names in CCR there is no automatic failover between interfaces.  Should the private interface serving log shipping be unavailable for any reason, log shipping will fail.  With this in mind appropriate monitoring of log copy operations is necessary to ensure replication is functioning.  In the event that the network link serving replication is not available, the host file should be removed and replication resumed over the public interface.  As mentioned earlier your network design considerations should take into account the need to communicate over both the public and private interfaces as well as the potential need to perform log shipping operations over the public interface. 

For the solution to be fully supported network connectivity must be available between the source and target on both the private and public interfaces.  All replication operations must be able to function on both interfaces.

When engaging product support services for assistance with replication when these steps are used you may be requested to remove the host file and verify that log shipping works as originally designed with no modifications.

Behavior of commandlets used for implementing / managing standby continuous replication when replication is enabled to use an alternate interface.

Get-storagegroupcopystatus:  No issues noted.

Enable-storagegroupcopy:  No issues noted.

Disable-storagegroupcopy:  No issues noted.

Restore-storagegroupcopy:  No issues noted when machines involved are running Exchange 2007 SP1 RU7.  Prior to RU7 it may be necessary to use restore-storagegroupcopy –force for the command to complete successfully.

Update-storagegroupcopy:  Because update-storagegroupcopy uses online streaming functionality to seed the database to the target the network traffic associated with this occurs over the public interface.

Suspend-storagegroupcopy:  No issues noted.

Resume-storagegroupcopy:  No issues noted.

Changes to the SCR activation process when replication is enabled to use an alternate interface.

Whether using the database portability method or the single node cluster method after running restore-storagegroupcopy the entries in the host file should be removed or commented out.  Once the removal is complete, dns resolver cache should be flushed (ipconfig /flushdns) and a ping from the target machine to it’s own name performed to ensure DNS resolves the correct IP address on the public interface.

When name resolution occurs successfully your move-mailbox –configurationonly or setup.com /recoverCMS can be run to complete the activation process.

**********************

Windows 2008

**********************

Configuring networks and network interfaces to support standby continuous replication using an alternate network interface on Windows 2008.

The first step is to configure the network settings for the network interface that will be used for standby continuous replication.  These instructions are performed on both the source and target machines.  To configure these settings:

• Select Start –> Control Panels –> Network and Sharing Center
• In the tasks pane, select “Manage Network Connections”
• Right click on the interface for SCR private replication, select rename.
  • Change the name of the interface to something meaningful indicating it’s purpose.
• Right click on the interface for SCR private replication, select properties.
  • Ensure that Client for Microsoft Networks is enabled.
  • Ensure the File and Print Sharing for Microsoft Networks is enabled.

• In the network properties, select “Internet Protocol Version 4 (TCP/IPv4)”, press properties button.
  • On the general tab:
    • Under “Use the following IP address”, provide a valid IP address and subnet mask.
    • Do not provide a default gateway.  (If the source and target reside on different subnets it will be necessary to utilize persistent static routes in order to ensure network communications follow the appropriate path.  Do not use two default gateways on a single host).
    • Do not provide any DNS servers under “Use the following DNS server addresses” .

• Selecting the advanced button, on the DNS tab:
  • Uncheck “Register this connection’s addresses in DNS”.

• Complete the changes by pressing OK three times.

The network configuration process is then completed by updating the network binding orders.  To update the network binding orders:

• Select Start –> Control Panels –> Network and Sharing Center
• In the tasks pane, select “Manage Network Connections”
• Ensure that the toolbar is displayed:
  • Select Organize –> Layout –> Menu Bar
• From the advanced menu select advanced settings.
• On the “Adapters and Bindings” tab, using the arrow keys, adjust the binding order so that the public facing interface is first on the list.  All other interfaces may be set in any order.

This completes the base networking configuration for standalone machines and clustered nodes.

Additional configuration steps for SCR source servers on Windows 2008.

• Exchange 2007 SP1 / Windows 2008 Standalone Mailbox Server Source
  • No additional configuration changes are necessary.
• Exchange 2007 SP1 / Windows 2008 Cluster Continuous Replication (CCR) Source
  • Windows 2008 enumerates all networks in a cluster for cluster use.  To ensure full supportability of the cluster, the cluster should be re-validated using the validation wizard in Failover Cluster Management.
  • No additional configuration changes are necessary.
• Exchange 2007 SP1 / Windows 2008 Single Copy Cluster (SCC) Source
  • Windows 2008 enumerates all networks in a cluster for cluster use.  To ensure full supportability of the cluster, the cluster should be re-validated using the validation wizard in Failover Cluster Management.
  • Enable the network for SCR replication for client connectivity.
    • Launch Failover Cluster Management
    • In the left hand pane, expand networks.
    • Identify the network that contains the interfaces used for standby continuous replication.
    • Right click on the network, select properties.
    • Select the checkbox next to “Allow clients to connect through this network”.
    • Press the OK button.
    • When prompted that the network is now available for client use, press the OK button.

• Enable an IP address resource in the Exchange Clustered Mailbox Server (CMS) group.
  • In the left hand pane, under services and applications, find the CMS group.
  • Right click on the group, select Add a Resource –> More Resources –> Add IP Address
  • This will create a not configured IP Address resource in the CMS group.

• Right click on the not configured IP Address resource and select properties.
• In the "Resource Name” field, enter and appropriate name.
• In the “Network” drop down, select the SCR replication network.
• In the “Static IP Address” fields input a static IP address valid on the SCR replication network.

• Press the OK button.
• Right click on the new IP address resource, select “Bring this resource online”.
• DO NOT UPDATE ANY DEPENDENCIES ON THIS RESOURCE OR MAKE THIS RESOURCE DEPENDANT ON ANY OTHER RESOURCE(S).

Additional configuration steps for SCR Targets on Windows 2008.

These instructions apply to both standalone and single node SCR targets based on Windows 2008.

Using notepad, open the hosts files located at c:WindowsSystem32DriversEtc

Depending on the source make the following changes:

• Exchange 2007 SP1 / Windows 2008 Standalone Mailbox Server Source
  • Add entries in the host file for both the NetBIOS and fully qualified domain name (FQDN).  Associate these names to the IP address assigned to the SCR network on the source server.
• Exchange 2007 SP1 / Windows 2008 Cluster Continuous Replication (CCR) Source
  • Add entries in the host file for both the NetBIOS names and fully qualified domain names (FQDN) of the source NODES.  Associate these names to the IP address assigned to the SCR network on the source nodes.
• Exchange 2007 SP1 / Windows 2008 Single Copy Cluster (SCC) Source
  • Add entries in the host file for both the NetBIOS name and fully qualified domain name (FQDN) of the source CMS.  Associate these names to the IP address created in the CMS group assigned to the SCR network.

Here is the output of a sample host file.

# Copyright (c) 1993-2006 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a ‘#’ symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

127.0.0.1       localhost
::1             localhost

#Exchange 2007 SP1 / Windows 2008 / Standalone Mailbox Server

10.1.1.1    2008-MBX1
10.1.1.1    2008-MBX1.exchange.msft

#Exchange 2007 SP1 / Windows 2008 / Cluster Continuous Replication (CCR)

10.1.1.3    2008-Node1
10.1.1.3    2008-Node1.exchange.msft
10.1.1.4    2008-Node2
10.1.1.4    2008-Node2.exchange.msft
10.1.1.8    2008-Node5
10.1.1.8    2008-Node5.exchange.msft
10.1.1.9    2008-Node6
10.1.1.9    2008-Node6.exchange.msft

#Exchange 2007 SP1 / Windows 2008 / Single Copy Cluster (SCC)

10.1.1.7    2008-MBX4
10.1.1.7    2008-MBX4.exchange.msft

Additionally, the replication service on occasion may have to resort to Netbios name resolution.  To ensure that the correct replication network is always returned, edit the LMHOST file and put entries for the netbios name and corresponding IP address.

Using notepad, open the LMhosts files located at c:WindowsSystem32DriversEtc

Here is a sample LMHost file.

# Copyright (c) 1993-1999 Microsoft Corp.
#
# This is a sample LMHOSTS file used by the Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to computernames
# (NetBIOS) names.  Each entry should be kept on an individual line.
# The IP address should be placed in the first column followed by the
# corresponding computername. The address and the computername
# should be separated by at least one space or tab. The "#" character
# is generally used to denote the start of a comment (see the exceptions
# below).
#
# This file is compatible with Microsoft LAN Manager 2.x TCP/IP lmhosts
# files and offers the following extensions:
#
#      #PRE
#      #DOM:<domain>
#      #INCLUDE <filename>
#      #BEGIN_ALTERNATE
#      #END_ALTERNATE
#      xnn (non-printing character support)
#
# Following any entry in the file with the characters "#PRE" will cause
# the entry to be preloaded into the name cache. By default, entries are
# not preloaded, but are parsed only after dynamic name resolution fails.
#
# Following an entry with the "#DOM:<domain>" tag will associate the
# entry with the domain specified by <domain>. This affects how the
# browser and logon services behave in TCP/IP environments. To preload
# the host name associated with #DOM entry, it is necessary to also add a
# #PRE to the line. The <domain> is always preloaded although it will not
# be shown when the name cache is viewed.
#
# Specifying "#INCLUDE <filename>" will force the RFC NetBIOS (NBT)
# software to seek the specified <filename> and parse it as if it were
# local. <filename> is generally a UNC-based name, allowing a
# centralized lmhosts file to be maintained on a server.
# It is ALWAYS necessary to provide a mapping for the IP address of the
# server prior to the #INCLUDE. This mapping must use the #PRE directive.
# In addtion the share "public" in the example below must be in the
# LanManServer list of "NullSessionShares" in order for client machines to
# be able to read the lmhosts file successfully. This key is under
# machinesystemcurrentcontrolsetserviceslanmanserverparametersnullsessionshares
# in the registry. Simply add "public" to the list found there.
#
# The #BEGIN_ and #END_ALTERNATE keywords allow multiple #INCLUDE
# statements to be grouped together. Any single successful include
# will cause the group to succeed.
#
# Finally, non-printing characters can be embedded in mappings by
# first surrounding the NetBIOS name in quotations, then using the
# xnn notation to specify a hex value for a non-printing character.
#
# The following example illustrates all of these extensions:
#
# 102.54.94.97     rhino         #PRE #DOM:networking  #net group’s DC
# 102.54.94.102    "appname  x14"                    #special app server
# 102.54.94.123    popular            #PRE             #source server
# 102.54.94.117    localsrv           #PRE             #needed for the include
#
# #BEGIN_ALTERNATE
# #INCLUDE \localsrvpubliclmhosts
# #INCLUDE \rhinopubliclmhosts
# #END_ALTERNATE
#
# In the above example, the "appname" server contains a special
# character in its name, the "popular" and "localsrv" server names are
# preloaded, and the "rhino" server name is specified so it can be used
# to later #INCLUDE a centrally maintained lmhosts file if the "localsrv"
# system is unavailable.
#
# Note that the whole file is parsed including comments on each lookup,
# so keeping the number of comments to a minimum will improve performance.
# Therefore it is not advisable to simply add lmhosts file entries onto the
# end of this file.

#Exchange 2007 SP1 / Windows 2008 / Standalone Mailbox Server

10.1.1.1    2008-MBX1

#Exchange 2007 SP1 / Windows 2008 / Cluster Continuous Replication (CCR)

10.1.1.3    2008-Node1
10.1.1.4    2008-Node2
10.1.1.8    2008-Node5
10.1.1.9    2008-Node6

#Exchange 2007 SP1 / Windows 2008 / Single Copy Cluster (SCC)

10.1.1.7    2008-MBX4

This completes the configuration steps for Windows 2008.

**********************

Windows 2003

**********************

Configuring networks and network interfaces to support standby continuous replication using an alternate network interface on Windows 2003.

The first step is to configure the network settings for the network interface that will be used for standby continuous replication.  These instructions are performed on both the source and target machines.  To configure these settings:

• Select Start –> Control Panels –> Network Connections
• Right click on the interface for SCR private replication, select rename
  • Change the name of the interface to something meaningful indicating it’s purpose.
• Right click on the interface for SCR private replication, select properties.
  • Ensure that Client for Microsoft Networks is enabled.
  • Ensure the File and Print Sharing for Microsoft Networks is enabled.

• In the network properties, select “Internet Protocol (TCP/IP)”, press properties button.
  • On the general tab:
    • Under “Use the following IP address”, provide a valid IP address and subnet mask.
    • Do not provide a default gateway.  (If the source and target reside on different subnets it will be necessary to utilize persistent static routes in order to ensure network communications follow the appropriate path.  Do not use two default gateways on a single host).
    • Do not provide any DNS servers under “Use the following DNS server addresses” .

• Selecting the advanced button, on the DNS tab:
  • Uncheck “Register this connection’s addresses in DNS”.

• Complete the changes by pressing OK three times.

The network configuration process is then completed by updating the network binding orders.  To update the network binding orders:

• Select Start –> Control Panels –> Network Connections
• From the advanced menu select advanced settings.
• On the “Adapters and Bindings” tab, using the arrow keys, adjust the binding order so that the public facing interface is first on the list.  All other interfaces may be set in any order.

This completes the base networking configuration for standalone machines and clustered nodes.

Additional configuration steps for SCR source servers on Windows 2003.

• Exchange 2007 SP1 / Windows 2003 Standalone Mailbox Server Source
  • No additional configuration changes are necessary.
• Exchange 2007 SP1 / Windows 2003 Cluster Continuous Replication (CCR) Source
  • No additional configuration changes are necessary.
• Exchange 2007 SP1 / Windows 2003 Single Copy Cluster (SCC) Source
  • Enable the network for SCR replication for client connectivity.
    • Launch Cluster Administrator
    • In the left hand pane, expand “Cluster Configuration” –> “Networks”.
    • Identify the network that contains the interfaces used for standby continuous replication.
    • Right click on the network, select properties.
    • Select “Enable this network for cluster use” and “All Communications (mixed network)”.  Press the OK button.

• Enable an IP address resource in the Exchange Clustered Mailbox Server (CMS) group.
  • In the left hand pane, expand groups, find the CMS group.
  • Right click on the group, select New –> Resource.
  • Provide an appropriate name in the Name field.
  • Provide an appropriate description in the Description field.
  • For Resource Type, change the drop down to “IP Address”

• Press the next button.
• Verify that the possible owners of the CMS group are correct.  Press the next button.

• DO NOT UPDATE ANY DEPENDENCIES ON THIS RESOURCE OR MAKE THIS RESOURCE DEPENDANT ON ANY OTHER RESOURCE(S).

• In the address field type an appropriate static IP address for the SCR replication network.
• In the subnet mask field type the appropriate subnet mask.
• In the network drop down ensure that the SCR replication network is selected.
• Keep “Enable NetBIOS for this address” selected.

• Press the finish button.
• Right click on the new IP address resource, select “Bring this resource online”.

Additional configuration steps for SCR Targets on Windows 2003.

These instructions apply to both standalone and single node SCR targets based on Windows 2003.

Using notepad, open the hosts files located at c:WindowsSystem32DriversEtc

Depending on the source make the following changes:

• Exchange 2007 SP1 / Windows 2003 Standalone Mailbox Server Source
  • Add entries in the host file for both the NetBIOS and fully qualified domain name (FQDN).  Associate these names to the IP address assigned to the SCR network on the source server.
• Exchange 2007 SP1 / Windows 2003 Cluster Continuous Replication (CCR) Source
  • Add entries in the host file for both the NetBIOS names and fully qualified domain names (FQDN) of the source NODES.  Associate these names to the IP address assigned to the SCR network on the source nodes.
• Exchange 2007 SP1 / Windows 2003 Single Copy Cluster (SCC) Source
  • Add entries in the host file for both the NetBIOS name and fully qualified domain name (FQDN) of the source CMS.  Associate these names to the IP address created in the CMS group assigned to the SCR network.

Here is the output of a sample host file.

# Copyright (c) 1993-2006 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a ‘#’ symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

127.0.0.1       localhost

#Exchange 2007 SP1 / Windows 2003 / Standalone Mailbox Server

10.1.1.1    2003-MBX1
10.1.1.1    2003-MBX1.exchange.msft

#Exchange 2007 SP1 / Windows 2003 / Cluster Continuous Replication (CCR)

10.1.1.3    2003-Node1
10.1.1.3    2003-Node1.exchange.msft
10.1.1.4    2003-Node2
10.1.1.4    2003-Node2.exchange.msft

#Exchange 2007 SP1 / Windows 2003 / Single Copy Cluster (SCC)

10.1.1.7    2003-MBX4
10.1.1.7    2003-MBX4.exchange.msft

Additionally, the replication service on occasion may have to resort to Netbios name resolution.  To ensure that the correct replication network is always returned, edit the LMHOST file and put entries for the netbios name and corresponding IP address.

Using notepad, open the LMhosts files located at c:WindowsSystem32DriversEtc

# Copyright (c) 1993-1999 Microsoft Corp.
#
# This is a sample LMHOSTS file used by the Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to computernames
# (NetBIOS) names.  Each entry should be kept on an individual line.
# The IP address should be placed in the first column followed by the
# corresponding computername. The address and the computername
# should be separated by at least one space or tab. The "#" character
# is generally used to denote the start of a comment (see the exceptions
# below).
#
# This file is compatible with Microsoft LAN Manager 2.x TCP/IP lmhosts
# files and offers the following extensions:
#
#      #PRE
#      #DOM:<domain>
#      #INCLUDE <filename>
#      #BEGIN_ALTERNATE
#      #END_ALTERNATE
#      xnn (non-printing character support)
#
# Following any entry in the file with the characters "#PRE" will cause
# the entry to be preloaded into the name cache. By default, entries are
# not preloaded, but are parsed only after dynamic name resolution fails.
#
# Following an entry with the "#DOM:<domain>" tag will associate the
# entry with the domain specified by <domain>. This affects how the
# browser and logon services behave in TCP/IP environments. To preload
# the host name associated with #DOM entry, it is necessary to also add a
# #PRE to the line. The <domain> is always preloaded although it will not
# be shown when the name cache is viewed.
#
# Specifying "#INCLUDE <filename>" will force the RFC NetBIOS (NBT)
# software to seek the specified <filename> and parse it as if it were
# local. <filename> is generally a UNC-based name, allowing a
# centralized lmhosts file to be maintained on a server.
# It is ALWAYS necessary to provide a mapping for the IP address of the
# server prior to the #INCLUDE. This mapping must use the #PRE directive.
# In addtion the share "public" in the example below must be in the
# LanManServer list of "NullSessionShares" in order for client machines to
# be able to read the lmhosts file successfully. This key is under
# machinesystemcurrentcontrolsetserviceslanmanserverparametersnullsessionshares
# in the registry. Simply add "public" to the list found there.
#
# The #BEGIN_ and #END_ALTERNATE keywords allow multiple #INCLUDE
# statements to be grouped together. Any single successful include
# will cause the group to succeed.
#
# Finally, non-printing characters can be embedded in mappings by
# first surrounding the NetBIOS name in quotations, then using the
# xnn notation to specify a hex value for a non-printing character.
#
# The following example illustrates all of these extensions:
#
# 102.54.94.97     rhino         #PRE #DOM:networking  #net group’s DC
# 102.54.94.102    "appname  x14"                    #special app server
# 102.54.94.123    popular            #PRE             #source server
# 102.54.94.117    localsrv           #PRE             #needed for the include
#
# #BEGIN_ALTERNATE
# #INCLUDE \localsrvpubliclmhosts
# #INCLUDE \rhinopubliclmhosts
# #END_ALTERNATE
#
# In the above example, the "appname" server contains a special
# character in its name, the "popular" and "localsrv" server names are
# preloaded, and the "rhino" server name is specified so it can be used
# to later #INCLUDE a centrally maintained lmhosts file if the "localsrv"
# system is unavailable.
#
# Note that the whole file is parsed including comments on each lookup,
# so keeping the number of comments to a minimum will improve performance.
# Therefore it is not advisable to simply add lmhosts file entries onto the
# end of this file.

#Exchange 2007 SP1 / Windows 2003 / Standalone Mailbox Server

10.1.1.1    2003-MBX1

#Exchange 2007 SP1 / Windows 2003 / Cluster Continuous Replication (CCR)

10.1.1.3    2003-Node1
10.1.1.4    2003-Node2

#Exchange 2007 SP1 / Windows 2003 / Single Copy Cluster (SCC)

10.1.1.7    2003-MBX4

This completes the configuration steps for Windows 2003.

===============================

Updated Sunday, August 9th, 2009 with LMHOST instructions.

===============================

When using any form of multi-machine Exchange 2007 replication (CCR / SCR) Kerberos authentication is very important.  We leverage the rights of Exchange server machine accounts for several functions including the ability to replicate log files and utilize the remote registry service for commandlets.

Some Background…

In terms of the replication service we copy logs between CCR and SCR servers using SMB file shares.  These shares are created by the replication service.  Permissions to access these shares are derived by assigning the share permission READ to the Exchange Servers group.

Note:  We use a very restrictive read heuristic which does not fully equal the same read permission as set through the GUI so you’ll have to trust me here – the groups effective share permission is read.

In the Exchange Servers group we automatically place the machine account for each Exchange server installed. 

By adding the Exchange servers machine account to the Exchange Servers group, and adding the Exchange Servers group with appropriate permissions, we’ve effectively allowed the machine accounts read access to the shares.  The replication service, which runs under the local system security context, can then access the shares to pull log files.

The Issue…

It is becoming more common in environments to no longer see WINS installations.  It is still a requirement of the product though that short name resolution work.  Many administrators address this issue by using a DNS suffix list.  You can set the DNS suffix list on the advanced settings of TCP/IP.

When a short name resolution request is made, the domains are appended in order as specified on the list.  If the name can be found in the DNS zone of the name as appended, then this combination is returned as the fully qualified domain name.

For example, if I have a host Server1 that is registered only in the DNS namespace exchange.msft, when I issue a ping request to Server1 the first domain appended is external.exchange.msft.  Since the machine is not registered in that dns domain, the second address is appended, in this case exchange.msft.  With the machine registered in this domain, a successful name query response is received and the ping will continue successfully to server1.exchange.msft.

An ipconfig /all will display the appended DNS suffixes and their order.

In many circumstances the machine will only resolve in a single DNS domain.  If this is the case then it will not affect keberos authentication.  Where an issue occurs is where the machine resolves in more then one domain, and the domain it resolves it does not match the active directory domain (which in this case is what is registered on the service principle name records) of the machine account.  Lets look at an example of how this can be an issue.

In this example the machine Server1 is registered in the dns domain external.exchange.msft and exchange.msft.  My active directory DNS name space for the domain that the machine is a member of is exchange.msft.

My append dns suffixes in this order has the following list:

external.exchange.msft

exchange.msft

domain.com

When the replication service attempts to copy a log file it begins the authentication process to the share.  The first step in this process is obtaining a kerberos ticket so we can leverage the permissions of the machine account (local system) for share access.  The first name in the suffix list is appended, and a successful name resolution occurs.  In this case the fully qualified domain name is believed to be Server1.external.exchange.msft.  At this time the kerberos key distribution center is contacted, and a ticket is issued for Server1.external.exchange.msft.  The next step is to access the share presenting this kerberos ticket.  At this time an access denied is received to the share, and logs cannot be copied. 

The reason the access denied is returned is that the service principle names stamped on the machine account in active directory for the server does not include Server1.external.exchange.msft, it only includes Server1.exchange.msft (the AD domain name).  You can see the SPNs registered on the server by doing an LDP dump of the computer object in the active directory domain container.  Here is an example:

servicePrincipalName (4): MSServerClusterMgmtAPI/2008-NODE1; MSServerClusterMgmtAPI/2008-Node1.exchange.msft; HOST/2008-Node1; HOST/2008-Node1.exchange.msft;

The issue in this case is easily corrected.  To correct the issue, change the appended DNS suffix list to use the active directory domain first.  For example:

exchange.msft

external.exchange.msft

domain.com

With the updated DNS suffix list the server name determined is server1.exchange.msft.  This name matches the entries of the service principle name and authentication can occur successfully and therefore log replication can occur without issues.

Other functions besides log replication can be impacted by the appended DNS suffix list.  For example, certain commandlets such as get-storagegroupcopystatus and update-storagegroupcopy leverage the rights of the local system to access the remote registry service.  These commandlets can also suffer access denied conditions as authenticated remote registry connections between servers can fail. 

Here is a sample of the error text from a failed get-storagegroupcopystatus:

Microsoft Exchange Replication service RPC failed :
Microsoft.Exchange.Rpc.RpcException: Error e0434f4d from cli_GetCopyStatusEx
at Microsoft.Exchange.Rpc.Cluster.ReplayRpcClient.GetCopyStatusEx(Guid[]
sgGuids, RpcStorageGroupCopyStatus[]& sgStatuses)
at
Microsoft.Exchange.Cluster.Replay.ReplayRpcClientWrapper.InternalGetCopyStatus(Strin
g serverName, Guid[] sgGuids, RpcStorageGroupCopyStatus[]& sgStatuses, Int32
serverVersion)
at
Microsoft.Exchange.Cluster.Replay.RpcCopyStatusInfo.GetMergedStatusResults()
at
Microsoft.Exchange.Management.SystemConfigurationTasks.GetStorageGroupCopyStatus.Pre
pareStatusEntryFromRpc(Boolean fCcr, Server server, StorageGroup storageGroup,
StorageGroupCopyStatusEntry& entry)”

The moral of the story…

Replication and commandlet issues on Exchange servers can be avoided when using appended dns suffixes list but ensuring that the active directory DNS domain is the first to be appended.

When attempting to establish the cluster services on nodes that utilize a dis-joint DNS namespace, the following errors may be encountered:

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date_Time
Event ID: 1127
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: ComputerName
Description:
Cluster Network interface InterfaceName for cluster node NodeName on network NetworkName failed. Run the Validate a Configuration wizard to check your network configuration.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date_Time
Event ID: 1207
Task Category: Network Name
Resource Level: Error
Keywords:
User: SYSTEM
Computer: Computer-name.domain.com Description: Cluster network name resource ‘Cluster Name’ cannot be brought online. The computer object associated with the resource could not be updated in domain ‘disjoined.domain.com’ for the following reason: Unable to update password for computer account.
The text for the associated error code is: The password does not meet the password policy requirements. Check the minimum password length, password complexity and password history requirements.
The cluster identity ‘Cluster-name$’ may lack permissions required to update the object. Please work with your domain administrator to ensure that the cluster identity can update computer objects in the domain.

If you see errors similar to this, check out the following two links that may apply.

http://technet.microsoft.com/en-us/library/cc755926(WS.10).aspx

http://support.microsoft.com/kb/952247/en-us

If you are using a Windows 2008 / Exchange 2007 Single Copy Cluster (SCC) you should read my co-workers blog post regarding disk resource is-alive checking.

http://blogs.technet.com/brian_kern/archive/2009/06/04/windows-2008-and-the-isalive-sanity-check.aspx

Tim

Recently there have been some questions about transaction log rolling and continuous replication.  In some cases these questions often surround storage group copy status showing an initializing state (http://blogs.technet.com/timmcmic/archive/2009/01/26/get-storagegroupcopystatus-initializing.aspx).

Under normal circumstances, the only time that log would roll, is when we’ve reached a log full condition.  If the server is being utilized, this is not a problem, as logs will roll naturally as the server processes activity.

There are times though where the server is relatively idle.  This would mean the current log generation would not receive enough transaction activity against it to cause it to roll over.  This is where “transaction log roll” is important.  If the current log file (ENN.log) contains a durable (or hard) commit, and that log is not filled in a period of time, it will be rolled over and shipped to the other side.  (This is not an immediate process, if we rolled a log over every time there was a durable (hard) commit we’d generate a ton of logs).  The article referenced above gives examples of how to calculate the time that a log would roll over should it contain a durable (hard) commit.  The article above also contains the following text highlighting this behavior:

“The log roll mechanism does not generate transaction logs in the absence of user or other database activity. In fact, log roll is designed to occur only when there is a partially filled log.”

This information is important to us for several reasons.

The first is generally if logs roll why do my storage groups stay initializing for hours at a time.  The answer is because the current log does not contain a durable commit.  If you were to restart the replication service or suspend and resume a replication instance manually the first replication state you will encounter is initializing.  We remain in initializing until a log is generated, copied, inspected, and put out for replay with divergence information determined.  If no durable (hard) commit exists in the source log stream, the logs may not be rolled over until there is a durable (hard) commit or user activity, which means replication would stay in an initializing state for a while.  My suggestion is, if this is a test environment, simply send mail / dismount the source databases / etc.  In production, I’ve seen people script email to test mailboxes at a schedule time with a test mailbox located in each database.  This causes a durable commit, which will eventually result in log file roll over and shipment to the other side.

The second reason is that log file roll can cause churn in the log file stream which does not appear normal.  If you reference the link above you can see that an idle storage group could generate up to 960 log files a day.  This is especially true of the storage group contains some type of system mailboxes (which exchange accesses causing a durable commit) or test mailboxes which the user is accessing.  In either scenario, there may not be enough load by either process to force log roll to occur naturally, so Exchange rolls the log for you at a certain time.  This causes some concern, especially when looking at the log file drive on a test server etc and questioning why so many logs were generated.  IE – there wasn’t enough traffic to generate 960 megs of logs, which is probably correct, but there was enough traffic to put a durable commit into each of those 960 logs such that we rolled and shipped them without being full in attempts to keep both sides up to date.

The third reason I pointed this out is that there seems to be confusion on when log roll should occur.  This leads to people believing the log roll should occur no matter what, when as indicated it should only occur if the log contains a durable (hard) commit. 

There are other operations besides user activity or a durable (hard) commit which will cause the current transaction log to roll:

• An attachment record is created in a log when a database is mounted.
• A VSS backup occurs of the active node.
• A VSS backup occurs of the passive node.
• An online streaming backup occurs of the active node.

I hope everyone finds this information helpful.

Recently there was a lively internal debate regarding how to use restore-storagegroupcopy and the –force switch.

The documentation regarding the restore-storagegroupcopy command can be found at http://technet.microsoft.com/en-us/library/aa996024.aspx.

According to the TechNet documentation:

“The Force parameter can be used when the task is run programmatically and prompting for administrative input is inappropriate. If Force is not provided in the cmdlet, administrative input is prompted. If Force is provided in the cmdlet, but the value is omitted, its default value is $true. When the Restore-StorageGroupCopy cmdlet is run to make an SCR target viable for mounting, the Force parameter must be included when the SCR source is not available.”

You’ll notice in this text that –force is required for standby continuous replication when the SCR source is not available.

So the first question is what constitutes the source being unavailable.  In the most general terms the source is unavailable when the shares where the log files reside are not available such that the restore-storagegroupcopy command can be run and the remaining logs copied between machines.

For Windows 2003 based sources, and Windows 2008 non-shared storage clusters, the shares are generally not available when the entire machine is offline.  For Windows 2008 shared storage clusters, the shares may not be available because their corresponding file server resources are offline in the clustered mailbox server group (for example, a stop-clusteredmailboxserver was issued taking the entire CMS offline, including the file server resources).  Of course there are other reasons that shares may not be available, like network issues / misc hardware issues / etc.

The reason I point this out is that if the source is available, and the –force command is being used, we will not copy the delta logs over to the SCR source and mark the databases mountable.  This effectively causes the database mount process to fail indicating log files necessary for recovery are not present.  Manual recovery using eseutil /r /a would have to be performed in order for the databases to mount.

The second question is how can I overcome this limitation so this does not happen to me?  The answer to that is simple.  If you run the restore-storagegroupcopy without the –force we will attempt to copy delta logs.  Should the source be unavailable, we will fail the copy procedure with a meaningful message indicating that the delta logs cannot be copied, and –force is necessary.  After receiving this error you can repeat the restore-storagegroupcopy, this time specifying the –force.  Since –force was required, the logs will not be copied (source unavailable) but the databases will be marked mountable.

Rule of Thumb:  First try restore-storagegroupcopy and only run restore-storagegroupcopy –force if indicated to do so in the error text of the command.

===========================================================

Example of successful activation using restore-storagegroupcopy where the shares are available (no –force used).

===========================================================

Environment:  Source cluster / target standalone.

The source clustered mailbox server was stopped using stop-clusteredmailboxserver.

An eseutil /ml of the source log directory was run, the end of the log file can be seen here.  You will see that the log stream is complete through the E01.log.

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000070.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000071.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000072.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000073.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000074.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000075.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000076.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 14.921 seconds.

Prior to running the restore-storagegroupcopy an eseutil /ml was run against the logs on the SCR target.  You will note that the same logs are present with the exception of the E01.log.  (This is expected, even when the source CMS is shutdown gracefully the last log in the series is not copied to the SCR target.)

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000070.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000071.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000072.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000073.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000074.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000075.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000076.log – OK

No damaged log files were found.

Operation completed successfully in 7.63 seconds.

At this time the shares on the source are available, and the mailbox stores dismounted.  A restore-storagegroupcopy –standbymachine <machine> is run and completes without error.  The following events are noted in the application log.

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/30/2009 8:20:16 AM
Event ID:      2114
Task Category: Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The replication instance for storage group MBX-2MBX-2-SG2 has started copying transaction log files. The first log file successfully copied was generation 119.

Log Name:      Application
Source:        MSExchangeRepl
Date:          4/30/2009 8:20:16 AM
Event ID:      2085
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The Restore-StorageGroupCopy operation on MBX-2MBX-2-SG2 was successful. All logs were successfully copied.

I then re-ran the eseutil /ml against the log series.  You will note that after the restore-storagegroupcopy –standbymachine:<machine> that the e01.log is now present, it was successfully copied as a part of the restore process.

I followed up with an eseutil /ml of the target log directory, you can now see that the E01.log is present in the directory.

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000071.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000072.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000073.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000074.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000075.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000076.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 7.250 seconds.

The last operation was to mount the databases.  At this time the databases mounted successfully – eseutil /r /a was not required.

Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          4/30/2009 8:25:06 AM
Event ID:      9523
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The Microsoft Exchange Database "MBX-3-SG2MBX-3-SG2-DB1" has been started.

Database File: G:MBX-2MBX-2-SG2-DatabaseMBX-2-SG2-DB1.edb
Transaction Logfiles: F:MBX-2MBX-2-SG2-Logs
Base Name (logfile prefix): E01
System Path: E:MBX-2MBX-2-SG2-System

===========================================================

===========================================================

Example of successful activation using restore-storagegroupcopy where the shares are not available (-force used).

===========================================================

Environment:  Source cluster / target standalone.

The clustered nodes comprising the source solution were completely shutdown making them completely unavailable.

Prior to shutting the nodes down, after issuing a stop-clusteredmailboxserver, and eseutil /ml was run against the log directory.  You will see the log stream is complete through E01.log.

Verifying log files…
     Base name: e01

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000092.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000093.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000094.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 0.78 seconds

Prior to running the restore-storagegroupcopy, an eseutil /ml was run against the logs on the SCR target.  You will note that the E01.log is not present.

Verifying log files…
     Base name: e01

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000092.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000093.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000094.log – OK

No damaged log files were found.

Operation completed successfully in 0.64 seconds.

At this time a restore-storagegroupcopy –standbymachine <MACHINE> was issued.  The following error was noted and expected since the source is no longer available.

[PS] G:>Restore-StorageGroupCopy -Identity MBX-2MBX-2-SG2 -StandbyMachine MBX-3
Restore-StorageGroupCopy : Restore failed to verify if the database on ‘MBX-2’ is mounted. Verify that the database is dismounted and then use the -Force parameter to restore the storage group copy.
At line:1 char:25
+ Restore-StorageGroupCopy  <<<< -Identity MBX-2MBX-2-SG2 -StandbyMachine MBX-3

After receiving an error that –force was necessary, the command was re-run using restore-storagegroupcopy –standbymachine –force.  The following information was presented in the Exchange Management Shell window:

[PS] G:>Restore-StorageGroupCopy -Identity MBX-2MBX-2-SG2 -StandbyMachine MBX-3    -force
WARNING: Performing a Restore-StorageGroupCopy operation on storage group
‘MBX-2-SG2’ with the Force option. Data loss is expected for this storage group.

The following events were noted in the application log:

Log Name:      Application
Source:        MSExchangeRepl
Date:          5/3/2009 10:37:39 AM
Event ID:      2139
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The forced Restore-StorageGroupCopy operation on MBX-2MBX-2-SG2 was successful. However, there may be some data loss.

After the command complete successfully, an eseutil /ml was performed against the log stream.  You will note that the e01.log is not present in the target log directory, since the remaining logs could not be copied due to the SCR source being unavailable.

Verifying log files…
     Base name: e01

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000092.log – OK 
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000093.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000094.log – OK

No damaged log files were found.

Operation completed successfully in 0.64 seconds.

At this time the database was successfully mounted as indicated by the following event in the application log.

Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          5/3/2009 10:44:06 AM
Event ID:      9523
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The Microsoft Exchange Database "MBX-3-SG2MBX-3-SG2-DB1" has been started.

Database File: G:MBX-2MBX-2-SG2-DatabaseMBX-2-SG2-DB1.edb
Transaction Logfiles: F:MBX-2MBX-2-SG2-Logs
Base Name (logfile prefix): E01
System Path: E:MBX-2MBX-2-SG2-System

===========================================================

===========================================================

Example of successful activation using restore-storagegroupcopy where the shares are available (-force used).

===========================================================

Environment:  Source cluster / target standalone.

The source clustered mailbox server was stopped using stop-clusteredmailboxserver.

An eseutil /ml of the source log directory was run, the end of the log file can be seen here.  You will see that the log stream is complete through the E01.log.

      Log file: F:MBX-2MBX-2-SG2-LogsE010000007A.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007B.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007C.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007D.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007E.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE01.log – OK

No damaged log files were found.

Operation completed successfully in 16.219 seconds.

Prior to running the restore-storagegroupcopy an eseutil /ml was run against the logs on the SCR target.  You will note that the same logs are present with the exception of the E01.log.  (This is expected, even when the source CMS is shutdown gracefully the last log in the series is not copied to the SCR target.)

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000079.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007A.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007B.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007C.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007D.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007E.log – OK

No damaged log files were found.

Operation completed successfully in 0.359 seconds.

At this time a restore-storagegroupcopy with the –force command was run.  Please note:  The source shares are available so –force is NOT NECESSARY.  Here is sample Exchange Management Shell output.

[PS] C:WindowsSystem32>Restore-StorageGroupCopy -Identity MBX-2MBX-2-SG2 –StandbyMachine MBX-3 –force

WARNING: Performing a Restore-StorageGroupCopy operation on storage group
‘MBX-2-SG2’ with the Force option. Data loss is expected for this storage
group.

The command completed successfully as indicated by returning to the Exchange Management Shell prompt without error.  The following event was noted in the application log.

Log Name:      Application
Source:        MSExchangeRepl
Date:          5/1/2009 8:29:41 AM
Event ID:      2139
Task Category: Action
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The forced Restore-StorageGroupCopy operation on MBX-2MBX-2-SG2 was successful. However, there may be some data loss.

As follow up eseutil /ml was run against the logs on the SCR target machine.  You will note that the E01.log was not copied even though the restore-storagegroupcopy –force command completed successfully.

      Log file: F:MBX-2MBX-2-SG2-LogsE0100000077.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000078.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE0100000079.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007A.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007B.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007C.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007D.log – OK
      Log file: F:MBX-2MBX-2-SG2-LogsE010000007E.log – OK

No damaged log files were found.

Operation completed successfully in 0.187 seconds.

At this time a database mount attempt was performed, and failed with the following events noted in the application log.

Log Name:      Application
Source:        MSExchangeIS
Date:          5/1/2009 8:32:13 AM
Event ID:      9518
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
Error Current log file missing starting Storage Group /DC=com/DC=domain/DC=domain/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=Organization/CN=Administrative Groups/CN=Exchange Administrative Group (FYDIBOHF23SPDLT)/CN=Servers/CN=MBX-3/CN=InformationStore/CN=MBX-3-SG2 on the Microsoft Exchange Information Store.
Storage Group – Initialization of Jet failed.

Log Name:      Application
Source:        ESE
Date:          5/1/2009 8:32:13 AM
Event ID:      455
Task Category: Logging/Recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
MSExchangeIS (2984) MBX-3-SG2: Error -1811 (0xfffff8ed) occurred while opening logfile f:MBX-2MBX-2-SG2-LogsE01.log.

The –1811 error translates to:

# for decimal -1811 / hex 0xfffff8ed
  JET_errFileNotFound
# /* File not found */
  JET_errFileNotFound
# /* File not found */
  JET_errFileNotFound
# /* File not found */
# 3 matches found for "-1811"

In this case the –force command was improperly used resulting in logs not being copied to the SCR target.  The databases could be mounted if they were manually recovered using eseutil /r /a or the logs manually copied to the SCR target.

This behavior is BY DESIGN.  The –force command does not check to see if the SCR source is available, therefore no log file copy attempts are made.

===========================================================