If an Exchange 2010 RTM server <or> an Exchange 2010 SP1 Beta has been upgraded to Exchange 2010 SP1 RTM administrators may experience an error when attempting to utilize the remove-mailboxdatabasecopy <or> add-mailboxdatabasecopy commandlets.

When running remove-mailboxdatabasecopy the following error is noted:

Remove-MailboxDatabaseCopy DAG-DB0DAG-2 –Verbose

WARNING: An unexpected error has occurred and a Watson dump is being generated: Registry key has subkeys and recursive removes are not supported by this method.

Registry key has subkeys and recursive removes are not supported by this method.
    + CategoryInfo          : NotSpecified: (:) [Remove-MailboxDatabaseCopy], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.SystemConfigurationTasks. 
  RemoveMailboxDatabaseCopy

Although the error is reported, the remove was successful in updating the database object within the active directory to show the server no longer hosts a copy of the database.  You can verify the copy was successfully removed by reviewing the Servers with the get-mailboxdatabase –identity <NAME> | fl name, servers commandlet.

Here is sample output (note DAG-2 is missing):

[PS] D:>Get-MailboxDatabase DAG-DB0 | fl name,servers

Name    : DAG-DB0
Servers : {DAG-1, DAG-3, DAG-4}

If an administrator attempts to add a database copy to a DAG member, the same error may also be returned.

Add-MailboxDatabaseCopy DAG-DB0 -MailboxServer DAG-2

WARNING: An unexpected error has occurred and a Watson dump is being generated: Registry key has subkeys and recursive
removes are not supported by this method.

Registry key has subkeys and recursive removes are not supported by this method.
    + CategoryInfo          : NotSpecified: (:) [Add-MailboxDatabaseCopy], InvalidOperationException
    + FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.Exchange.Management.SystemConfigurationTasks.
   AddMailboxDatabaseCopy

Unlike the remove-mailboxdatabasecopy this command is not successful in adding the copy <or> updating the Active Directory to show the copy was added.

To work around this issue the administrator should:

1)  Identify the GUID of the database that is being added.

2)  On the server specified in the add command, using the database GUID identified, remove the following registry key:

HKEY_LOCAL_MACHINESOFTWAREMicrosoftExchangeServerv14ReplayState{DB-GUID}DumpsterInfo

To identify the mailbox database GUID, use the following command:

[PS] D:>Get-MailboxDatabase DAG-DB0 | fl name,GUID

Name : DAG-DB0
Guid : 8d3a9778-851c-40a4-91af-65a2c487b4cc

The GUID identified in this case is 8d3a9778-851c-40a4-91af-65a2c487b4cc.  With this information we can no export and delete the DUMPSTERINFO key on the server where you are attempting to add the mailbox database copy.

Once the registry key is removed the add-mailboxdatabasecopy command will complete successfully and the database copy will be added.

 

A question that has come up a few times recently is why does the date modified timestamp on my Exchange databases not change (even though the database is mounted and functioning).  Specifically some administrators have been looking at this as an indicator of health on a passive database copy – which it is not.

The date modified timestamp will generally get updated on an Exchange database when one of two things happen:

1)  The EDB file size is extended in order to accommodate data that does not fit into whitespace that currently exists in the database.

2)  The database is dismounted and all open handles to the file are released.

Note that the modified time is not subject to change if the contents of the file are changed – for example if whitespace is utilized within the database for the storage of new messages etc the date modified will not change.

To show this I used my lab to generate some examples.  Here is a screen shot of a database that was mounted last on 8/3/2010.  The database screen shot was taken 8/8/2010 before 8:29 am edt.

 

Using the Exchange Management Console, I dismounted the database at 8:29 am edt on 8/8/2010.

You will note that the date modified changed to the time and date the dismount occurred.  I then used the Exchange Management Console to re-mount the database.

After remounting the database I noted that the time remained the same as in the previous screen shot.  I then took some test mailboxes with content, and moved them into the mailbox store.  You will note in this screen shot that both the size and date modified changed – in this case the database file was extended on the partition so the change was expected.

It is normal for an Exchange database to not show an updated date modified and this field should be used to judge the health or utilization of an Exchange database.

In Exchange 2007 there are two clustered installation models.  Some customers elect to utilize a clustered installation model based on shared storage – this is a single copy cluster installation.  In order to achieve site resiliency or provide for disaster recovery, some customers will implement a SAN based data replication solution. 

Recently I encountered a customer that was utilizing SAN based data replication and the single copy cluster installation model to provide their site resilient solution.  The installation encompassed a source cluster with single copy configuration and a target cluster with single copy cluster configuration.  Each clustered mailbox server was established utilizing a different name – for example Exchange-Main and Exchange-DR.  The physical disk resources that were assigned to each CMS instance represented the LUNs that were replicated between SANs.  When it was necessary to activate the solution databases would be marked as “Allow this database to be overwritten by a restore” and then mounted.  Mailboxes would be moved utilizing the move-mailbox –configurationOnly to restore client access to the replicated databases

This presented an interesting challenge for this customer when it came to deploying service packs.  When the same physical disk resources are utilized between clusters, only one set of the physical disk resources can be brought online.  This is because one SAN has a Read / Write setting and the other SAN has a Read Only setting.  Essentially an online attempt of the database instances of the CMS Exchange-DR would fail because their dependant physical disks could not be brought online (because they were read only).

When an /upgradeCMS is performed after upgrading the binaries on a clustered node, the resources are initially in an offline state.  As a completion of the upgradeCMS the setup process initiates an online to the cluster mailbox server group.  Should any resources fail to come online this is considered a failure of the upgrade.  The administrator performing the upgrade is notified that a failure occurred and the upgrade setup watermark persists in the registry.  Therefore it is necessary that the /upgradeCMS be allowed to complete.  In this case database instances could not be brought online because their associated storage could not be brought online due to the storage being Read Only.

In order to complete the upgrade process the following steps were utilized (utilizing my sample clustered mailbox server names).

• Following SAN vendor recommendations replication was suspended between the Exchange-Main and Exchange-DR. 
• Mark LUNs on the remote SAN as Read / Write (allowing Exchange-DR full access to storage).
• Databases on the secondary CMS were set to “Allow this database to be overwritten by a restore”
  • Get-MailboxDatabase –server Exchange-DR | Set-MailboxDatabase –allowfilerestore:$TRUE
• Complete the upgrade process on Exchange-Main which will fully bring resources online.
• Complete the upgrade process on Exchange-DR which will fully bring resources online.

At this point both Exchange-Main and Exchange-DR are online.  This means that the databases that were previously replicated to Exchange-DR are no longer equal to the databases that exist on Exchange-Main.  As a post upgrade step we need to do the following:

• Stop the resources on Exchange-DR.
• Mark replicated LUNS on the remote SAN as Read Only (preventing Exchange-DR access to the storage).
• Following SAN vendor recommendations re-establish replication between the source and remote SANs ensuring that the SOURCE SAN is utilized for data synchronization.

In this installation it was necessary to temporarily break and re-establish replication in order to complete the /upgradeCMS process.