Recently I’ve worked with several Exchange 2007 customers that are leveraging storage replication solutions with a Single Copy Cluster (SCC) as part of their site resiliency / disaster recovery solution for Exchange data. As a part of these implementations, customers are pre-staging clusters in their standby datacenters <and> creating Exchange clustered resources for these clusters.

In general, two configurations are typically seen:

1. The same clustered mailbox server (CMS) is recovered to a standby cluster.

2. An alternate CMS is installed and mailboxes are moved to the standby cluster.

In part 1 of this series, I will address the first method –recovering the original CMS to a standby cluster.

In part 2 of this series, I will address the second method.

First, let’s take a look at the topology.

In my primary site, I establish a two-node shared-storage cluster with NodeA and NodeB. In my remote datacenter, I establish a second two-node shared-storage cluster with NodeC and NodeD. Third-party storage replication technology is used to replicate the storage from the primary site to the remote site.

 

Figure 1 – Implementation prior to introduction of CMS

 

On the primary cluster, I install CMS named MBX-1 in an SCC configuration and create my desired storage groups and databases. This in turn creates the associated cluster resources for the database instances (in Exchange 2007, each database has an associated clustered resource called a Microsoft Exchange Database Instance).

From a storage standpoint, the disks connected to the primary cluster are in read-write mode and the disks connected to the standby cluster are in read-only mode.

 

Figure 2 – Implementation after introduction of CMS in primary site

 

Figure 3 – Example of database instances as seen in failover cluster manager

 

After preparing the CMS in the primary site, the administrator prepares the secondary site. As part of this preparation, the existing CMS is taken offline. Then, the administrator changes the replication direction of the storage, making the storage connected to the standby cluster R/W and the storage connected to the primary cluster R/O. Both storage solutions are synchronized so that they contain same the data.

Once storage synchronization has completed the administrator uses the /recoverCMS process to recover MBX-1 to the standby cluster. The /recoverCMS process reads the CMS configuration data from Active Directory and then recreates the CMS and its resources on the standby cluster.

 

Figure 4 – Implementation after introduction of the CMS in the remote site

 

At this point the same CMS exists on two different clusters. After the standby CMS has been brought online and validated on the standby cluster, the CMS is moved back to the primary cluster and the direction of storage replication is again reversed. The storage connected to the primary cluster is in R/W mode and the storage connected to the standby cluster is in R/O mode.

Once storage synchronization has completed the administrator brings the CMS on the primary cluster online.

Next, the administrator updates the RedundantMachines property of the CMS to reflect the nodes in the primary cluster.

 

Figure 5 – Implementation after introduction of CMS in the remote site and activation of CMS in the primary site

 

Because these solutions are often used for site resilience, when a failure of the primary cluster or site occurs, the administrator will perform the following steps to activate the standby cluster.

· Ensure all CMS resources are offline on the primary site cluster

· Change storage from R/O to R/W in the remote site

· Update the redundantMachines property to reference the nodes in the standby cluster

· Bring the CMS online on the remote servers

Often these steps work just fine without any issues. But recently I’ve worked on some cases where this process does not work.

Let’s take a look at some issues that may arise with this type of implementation.

1. Exchange was not designed to have the same resources exist simultaneously on two different clusters. Any recovery using pre-staged resources is not a recommended recovery mechanism for Exchange servers (we’ll talk about the recommeded recovery process shortly).

 

2. Administrators sometimes fail to update the redundantMachines attribute of the CMS. Each CMS has a property called redundantMachines. This property is a list of the names of nodes that can take ownership of the CMS. In general, the /recoverCMS process will reset this property for a CMS when the CMS is recovered to a different set of nodes.

In this case, the resources are pre-staged and /recoverCMS is not used after the initial configuration. As a result, the administrator must manually set this property using the Set-MailboxServer cmdlet. If an administrator fails to do this, other cmdlets that depend on this attribute (like Start-ClusteredMailboxServer, Move-ClusteredMailboxServer and Stop-ClusteredMailboxServer) will fail.

 

3. Resource configuration on the standby cluster is static.

Each database on a CMS has an associated clustered resource. When pre-staging the standby cluster, you are copying the configuration that existed at that time. Often, the configuration of the CMS on the primary cluster will change over time. I have worked with customers who added storage groups and databases to a CMS to a primary cluster after the standby cluster was configured. This results in clustered resources missing from the standby cluster.

To resolve this problem, some administrators have attempted to manually create clustered resources for the missing database instances. Unfortunately, this is not supported, and it results in the administrator having to follow a process similar to the one I recommend below.

 

4. Issues when applying Exchange Service Packs

When applying Exchange service packs to a CMS, the final step is to run /upgradeCMS. In order for /upgradeCMS to be considered successful (which is defined as the upgrade process reporting success and the CMS watermark being cleared from the registry) all of the resources on the cluster must be brought online.

For the primary cluster this does not present any issues. However, it is an issue for the standby cluster. On the standby cluster the following resources will not be able to come online:

· Physical Disk Resources – these resources in the remote site cluster are R/O and cannot brought online for the cluster upgrade

· Network Name Resource – this would result in a duplicate name on the network

Therefore, /upgradeCMS will fail. To resolve this condition, an administrator must either take the primary cluster offline or isolate the standby cluster from the primary cluster in order to complete the upgrade.

 

 

Obviously, this process could cause some longer term issues in the environment after its initial establishment. So, I want to outline a process that I’ve recommended in these environments. The first few parts of the process are the same as above:

1. In my primary site, I establish a two-node shared-storage cluster with NodeA and NodeB. In my remote datacenter, I establish a second two-node shared-storage cluster with NodeC and NodeD. Third-party storage replication technology is used to replicate the storage from the primary site to the remote site.

 

Figure 6 – Implementation prior to introduction of CMS

 

2.  On the primary cluster, I install CMS named MBX-1 in an SCC configuration and create my desired storage groups and databases. This in turn creates the associated cluster resources for the database instances.

3.  From a storage standpoint, the disks connected to the primary cluster are in read-write mode and the disks connected to the standby cluster are in read-only mode.

 

Figure 7 – Implementation after introduction of CMS in primary site

 

4.  On the standby cluster I prepare each node by installing and configuring the SCC, but instead of performing a /recoverCMS operation, I install only the passive mailbox server role on each node. This is done by running setup.com /mode:install /roles:mailbox. This process puts the Exchange program files on the system, performs cluster registrations, and prepares the nodes to accept a CMS at a later time.

 

Figure 8 – Implementation after introduction of CMS in primary site and passive role installation on clustered nodes in remote site

 

At this point, all preparation for the two sites is completed. When a failure occurs and a decision is made to activate the standby cluster I recommend that customers use the following procedure:

 

1.  Ensure that all CMS resources on the primary cluster are offline.

2.  Change the replication direction to allow the disks in the remote site to be R/W and the disks in the primary site to be R/O.

 

Figure 9 – Storage in remote site changed to R/W

 

3. Use the Exchange installation media to run the /recoverCMS process and establish the CMS on the standby cluster.

setup.com /recoverCMS /cmsName:<NAME> /cmsIPV4Addresses:<IPAddress,IPAddress>

Figure 10 – CMS recovery to passive nodes in remote site.

 

4. Move disks into appropriate groups and update resource dependencies as necessary.

At this point, the resources have been established on the standby cluster and clients should be able to resume connectivity.

 

Assuming that the primary site will come back up and the original nodes are available, the following process can be used to prepare the nodes in the primary site.

1. Ensure that the disks and network name do not come online. This can be accomplished by ensuring that the nodes have no network connectivity.

2. On the node that shows as owner of the offline Exchange CMS group, run the command setup.com /clearLocalCMS. The setup command will clear the local cluster configuration from those nodes and remove the CMS resources. The physical disk resources will be maintained in a cluster group that was renamed.

 

Figure 11 – Removal of the CMS in the source site.

 

3.  Ensure that storage replication is in place, healthy, and that a full synchronization of changes has occurred.

4.  Schedule downtime to accomplish the failback to the source nodes.

During this downtime, use the following steps can be utilized to establish services in the primary site.

 

1.  Take the CMS offline in the remote site.

 

Figure 12 – CMS offline in remote site.

 

2.  On the node owning the Exchange resource group in the remote site cluster execute a setup.com /clearLocalCMS command.  This will remove the clustered instance from the remote cluster.

 

Figure 13:  Removal of the CMS resources from the remote site cluster.

 

3.  Change the replication direction to allow the disks in the primary site to be R/W and the disks in the remote site to be R/O.

 

Figure 14:  Disks in primary site changed to R/W.  Disks in remote site changed to R/O

 

4.  Using setup media run the /recoverCMS command to establish the clustered resources on the standby cluster.

setup.com /recoverCMS /cmsName:<NAME> /cmsIPV4Addresses:<IPAddress,IPAddress>

 

Figure 15:  Recovery of CMS resources completed to primary site cluster.

 

5. Move disks into appropriate groups and update dependencies as necessary. 

6.  Clients should be able to resume connectivity when this process is completed.

 

How does this address the issues that I’ve outlined above?

1. The /recoverCMS process is a fully supported method to recover a CMS between nodes.

2. The /recoverCMS process is responsible for updating the redundantMachines property of the CMS. This prevents the administrator from having to manually change this as resources are recovered between clusters.

3. The /recoverCMS process will always recreate resources based on the configuration information in the directory. If databases are added to the primary cluster, the appropriate resources will be populated on the standby cluster when /recoverCMS is run. Similarly, if the CMS runs on the standby cluster for an extended period of time, and additional resources are created there, they will be added to the primary cluster when it is restored to service.

4. Service pack upgrades can be performed without having any special configuration. On the primary cluster you follow the standard practice of upgrading the program files with setup.com /mode:upgrade and then upgrading the CMS using setup.com /upgradeCMS. The nodes in the standby cluster are independent passive role installations and can be upgraded by using setup.com /mode:upgrade.

Recently I worked with a customer that was having backup failures when attempting to backup passive database copies on an Exchange 2010 DAG member.  Active database copies would backup without any issues.

 

The issue reproduced with both the commercial VSS product and utilizing VSS test procedures with the DISKSHADOW utility.

 

When reviewing the application log at the time of the backup the following events were noted:

 

Time:     6/28/2011 10:34:49 AM
ID:       2021
Level:    Information
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup.

Time:     6/28/2011 10:35:15 AM
ID:       9606
Level:    Information
Source: MSExchangeIS
Machine:  server.company.com
Message:  Exchange VSS Writer (instance 0afd4825-b904-4bf0-87ee-93568351c4ca) has prepared for backup successfully.

Time:     6/28/2011 10:35:16 AM
ID:       2110
Level:    Information
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange VSS Writer instance 0afd4825-b904-4bf0-87ee-93568351c4ca has successfully prepared for a full or a copy backup of database ‘nambx1-old’.  The following database will be backed up: <DATABASE>.

Time:     6/28/2011 10:35:16 AM
ID:       2023
Level:    Information
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange Replication service VSS Writer (Instance 0afd4825-b904-4bf0-87ee-93568351c4ca) successfully prepared for backup.

Time:     6/28/2011 10:35:17 AM
ID:       2021
Level:    Information
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange VSS Writer has successfully collected the metadata document in preparation for backup.

Time:     6/28/2011 10:35:55 AM
ID:       9539
Level:    Information
Source: MSExchangeIS Mailbox Store
Machine:  server.company.com
Message:  The Microsoft Exchange Information Store database "b79d42eb-c574-4ebb-8467-b3d0ec166817: /o=Organization/ou=Exchange Administrative Group(FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=server.company.com/cn=Microsoft Private MDB" was stopped.

Time:     6/28/2011 10:37:05 AM
ID:       2027
Level:    Information
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange VSS Writer instance 0afd4825-b904-4bf0-87ee-93568351c4ca has successfully frozen the databases.

Time:     6/28/2011 10:37:26 AM
ID:       2026
Level:    Error
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange Replication service VSS Writer (Instance 0afd4825-b904-4bf0-87ee-93568351c4ca) failed with error C7FF07D7 when preparing for snapshot.

Time:     6/28/2011 10:37:26 AM
ID:       8229
Level:    Warning
Source: VSS
Machine:  server.company.com
Message:  A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester.
Check the event log for related events from the application hosting the VSS writer.

Operation:
   PrepareForSnapshot Event

Context:
   Execution Context: Writer
   Writer Class Id: {76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}
   Writer Name: Microsoft Exchange Replica Writer
   Writer Instance Name: Exchange Replication Service
   Writer Instance ID: {a31bfcaa-668f-4a81-9cde-f9dfa2cadd5a}
   Command Line: "C:Program FilesMicrosoftExchange ServerV14binmsexchangerepl.exe"
   Process ID: 3972

Time:     6/28/2011 10:37:26 AM
ID:       2031
Level:    Information
Source: MSExchangeRepl
Machine:  server.company.com
Message:  The Microsoft Exchange Replication service VSS Writer (Instance 0afd4825-b904-4bf0-87ee-93568351c4ca) has successfully terminated the snapshot.

The event sequence essentially told us that we were to the point where we wanted to invoke the freeze of the database.  There was a failure directly before this process that caused the replication service VSS writer to abort the backup.  This abortion was in turn returned to the VSS framework and the passive copy backup cleaned up.

 

I specifically focused on the event 2026 with the error C7FF07D7.  While researching I noticed that other products and components also produced C7FF07D7 errors.  In these cases the error was returned when an RPC call between services failed – and a common theme was a networking or connectivity issue.

 

With this information in hand I started to run generic ping tests between the nodes to verify connectivity / dropped packets / etc.  This is where the breakthrough on this particular issue came out.  When pinging the nodes by netbios name the output looked as follows:

 

Pinging NODE [W.X.Y.Z] with 32 bytes of data:
Reply from W.X.Y.Z: bytes=32 time=3ms TTL=128
Reply from W.X.Y.Z: bytes=32 time<1ms TTL=128
Reply from W.X.Y.Z: bytes=32 time<1ms TTL=128
Reply from W.X.Y.Z: bytes=32 time<1ms TTL=128

Ping statistics for W.X.Y.Z:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 3ms, Average = 0ms

 

When pinging the netbios name of an Exchange server we expect that the fully qualified domain would be appended.  Generally this is appended from the DNS search suffix list (if specified) or the AD domain the server is a member of.  In this case there was no domain name appended to the server name.  This either points to an issue with the DNS search suffix list (which was populated appropriately and therefore not our problem) or an entry in the host file.

 

When reviewing the host file the following contents were noted:

 

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a ‘#’ symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#    127.0.0.1       localhost
#    ::1             localhost

W.X.Y.Z    NODE

 

At or near the time that the backup issue started occurring the host files on the nodes were modified to include an entry for the members of the DAG.  The entries in the host file only included the netbios name of the members, and did not include the name in fully qualified domain name format.   Once the entry was removed from the host file, dns resolver cache flushed, a ping test was issued and the expected results displayed.

 

Pinging NODE.COMPANY.COM [W.X.Y.Z] with 32 bytes of data:
Reply from W.X.Y.Z: bytes=32 time=3ms TTL=128
Reply from W.X.Y.Z: bytes=32 time<1ms TTL=128
Reply from W.X.Y.Z: bytes=32 time<1ms TTL=128
Reply from W.X.Y.Z: bytes=32 time<1ms TTL=128

Ping statistics for W.X.Y.Z:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 3ms, Average = 0ms

When a passive copy backup is performed (a surrogate backup) there is certain mandatory information that must be exchanged between the replication service on the passive copy and the information store service on the active copy.  This information is exchanged prior to freezing the database to service the snapshot.  If for any reason this information cannot be exchanged the replication service will abort the VSS backup and subsequently the backup will fail.  In this case name resolution between the nodes not working as expected caused this connection to fail and the information exchange to fail.  This prevented passive copy backups from being successful.

In Exchange 2010 Service Pack 1 we introduced some new DAG management scripts. These scripts can be found in the Exchange Server installation directory scripts. (This is usually c:Program FilesMicrosoftExchange Serverv14scripts).

 

One of the scripts introduced is the StartDagServerMaintenance.ps1 script. More information on this script can be found at:

http://technet.microsoft.com/en-us/library/ff625233.aspx

http://technet.microsoft.com/en-us/library/dd298065.aspx

 

When administrators utilize this script the following actions are being taken:

1) All database copies are moved to another server in the DAG based on the selection of the next best copy.

2) If the cluster core resources are owned on the node the resources are arbitrated to a different DAG member (thereby moving the Primary Active Manager functionality to another node).

3) The DatabaseCopyAutoActivationPolicy property of the mailbox server is set to a value of BLOCKED thereby preventing the DAG member from receiving or activating database copies.

4) The individual database copies hosted on the DAG member are activation suspended.

5) The node is paused within the cluster service preventing the cluster core resources from arbitrating to the node (and thereby preventing the node from becoming the Primary Active Manager).

 

When an administrator attempts to place a DAG member into maintenance mode and the DAG member hosts an ACTIVE database that has only two copies the following occurs:

1)  The database copy is moved to the other node hosting the passive copy (pending the copy is healthy).

2)  The command fails with the following error after the database is moved.  (In this example the mounted copy is on server DAG-4).

 

*Pre StartDagServerMaintenance*

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————

TESTSCRIPTDAG-4                              Mounted         0         0                                  Healthy

TESTSCRIPTDAG-3                              Healthy         0         0           7/25/2011 10:17:30 AM  Healthy

*StartDagServerMaintenance*

 

[PS] C:Program FilesMicrosoftExchange ServerV14Scripts>.StartDagServerMaintenance.ps1 DAG-4
The following objects are hosted by ‘DAG-4′, before attempting to move them off: `n(Database=’TESTSCRIPT’, Reason=’Copy is active’))
Write-Error : The following objects are still hosted by ‘DAG-4′, even after attempting to move them off: `n(Database=’TESTSCRIPT’, Reason=’Copy is critical for redundancy according to Red Alert script’))
At C:Program FilesMicrosoftExchange ServerV14ScriptsStartDagServerMaintenance.ps1:216 char:16
+                 write-error <<<<  ($StartDagServerMaintenance_LocalizedStrings.res_0014 -f ( PrintCriticalMailboxResourcesOutput($criticalMailboxResources)),$shortServerName) -erroraction:stop
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Microsoft.PowerShell.Commands.WriteErrorCommand

*Post StartDagServerMaintenance*

 

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
TESTSCRIPTDAG-3                              Mounted         0         0                                  Healthy
TESTSCRIPTDAG-4                              Healthy         0         0           7/25/2011 10:33:57 AM  Healthy

When an administrator attempts to place a DAG member into maintenance mode and the DAG member hosts an PASSIVE database that has only two copies the following occurs:

1) The command fails with the following error after the database is moved. (In this example the passive copy is on server DAG-4).

 

*Pre StartDagServerMaintenance*

 

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
TESTSCRIPTDAG-3                              Mounted         0         0                                  Healthy
TESTSCRIPTDAG-4                              Healthy         0         0           7/25/2011 10:33:57 AM  Healthy

 

*StartDagServerMaintenance*

 

[PS] C:Program FilesMicrosoftExchange ServerV14Scripts>.StartDagServerMaintenance.ps1 DAG-4
The following objects are hosted by ‘DAG-4′, before attempting to move them off: `n(Database=’TESTSCRIPT’, Reason=’Copy is active’))
Write-Error : The following objects are still hosted by ‘DAG-4′, even after attempting to move them off: `n(Database=’TESTSCRIPT’, Reason=’Copy is critical for redundancy according to Red Alert script’))
At C:Program FilesMicrosoftExchange ServerV14ScriptsStartDagServerMaintenance.ps1:216 char:16
+ write-error <<<< ($StartDagServerMaintenance_LocalizedStrings.res_0014 -f ( PrintCriticalMailboxResourcesOutput($criticalMailboxResources)),$shortServerName) -erroraction:stop
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Microsoft.PowerShell.Commands.WriteErrorCommand

 

*Post StartDagServerMaintenance*

 

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
TESTSCRIPTDAG-3                              Mounted         0         0                                  Healthy
TESTSCRIPTDAG-4                              Healthy         0         0           7/25/2011 10:33:57 AM  Healthy

Administrators can find manual maintenance mode instructions available in the following blog post:

http://blogs.technet.com/b/timmcmic/archive/2011/07/25/exchange-2010-sp1-startdagservermaintenance-ps1-fails-when-a-server-contains-databases-with-a-single-copy.aspx

 

After completing the manual instructions and when maintenance mode is no longer needed the administrator may utilize the StopDagServerMaintenance.ps1 script to revert the manual changes.

Recently I had the opportunity to work on an Exchange 2010 Database Availability Group installation where nodes were installed in multiple data centers.  Although not uncommon to have nodes installed in multiple data centers what was uncommon was the number of data centers and their geographical dispersion  – in this case 5 different data centers.  To make the environment even more challenging fully meshed networks were not deployed.  It was this factor in the design that caused some unexpected failover conditions when certain network links were lost.

 

The following article written by my co-worker Jeff Hughes – from our platforms team – addresses failover cluster service handling of this type of failure.

 

Happy reading!

 

http://blogs.technet.com/b/askcore/archive/2011/08/08/partitioned-cluster-networks.aspx

http://blogs.technet.com/b/jamec/archive/2011/08/01/exchange-2010-database-page-fragmentation-caused-by-archive-solutions.aspx

In Exchange 2010 Service Pack 1 we introduced some new DAG management scripts.  These scripts can be found in the Exchange Server installation directory scripts.  (This is usually c:Program FilesMicrosoftExchange Serverv14scripts).

 

One of the scripts introduced is the StartDagServerMaintenance.ps1 script.  More information on this script can be found at:

 

http://technet.microsoft.com/en-us/library/ff625233.aspx

http://technet.microsoft.com/en-us/library/dd298065.aspx

 

When administrators utilize this script the following actions are being taken:

1)  All database copies are moved to another server in the DAG based on the selection of the next best copy.

2)  If the cluster core resources are owned on the node the resources are arbitrated to a different DAG member (thereby moving the Primary Active Manager functionality to another node).

3)  The DatabaseCopyAutoActivationPolicy property of the mailbox server is set to a value of BLOCKED thereby preventing the DAG member from receiving or activating database copies.

4)  The individual database copies hosted on the DAG member are activation suspended.

5)  The node is paused within the cluster service preventing the cluster core resources from arbitrating to the node (and thereby preventing the node from becoming the Primary Active Manager).

 

When utilizing a DAG it is not necessary to replicate all databases that exist on DAG members.  It is not uncommon to have standalone databases (databases that are on a DAG member but not replicated to another member) present on a member where the StartDagServerMaintenance.ps1 script will be utilized.  Unfortunately when utilizing the script in its current form in this configuration the script fails to complete its tasks and cannot completely put the node into maintenance mode.   (Only databases are successfully moved off the member).

 

The administrator may note the following when executing the script on a member that contains a single database copy:

 

[PS] C:Program FilesMicrosoftExchange ServerV14Scripts>.StartDagServerMaintenance.ps1 -serverName DAG-1

The following objects are hosted by ‘DAG-1′, before attempting to move them off: `n(Primary Active Manager=DAG-1) (Mailbox=’Discovery Search Mailbox’, Reason=’Mailbox is hosted on ‘DAG-1-DB0′, which is not a replicated database. ) (Mailbox=’Journal Internal’, Reason=’Mailbox is hosted on ‘DAG-1-DB0′, which is not a replicated database. ) (Mailbox=’MicrosoftExchange Approval Assistant’, Reason=’Arbitration Mailbox is hosted on ‘DAG-1-DB0′, which is not a replicated database.) (Database=’DAG-DB0′, Reason=’Copy is active’))

Write-Error : The following objects are still hosted by ‘DAG-1′, even after attempting to move them off: `n(Mailbox=’Discovery Search Mailbox’, Reason=’Mailbox is hosted on ‘DAG-1-DB0′, which is not a replicated database. ) (Mailbox=’Journal Internal’, Reason=’Mailbox is hosted on ‘DAG-1-DB0′, which is not a replicated database. ) (Mailbox=’Microsoft Exchange Approval Assistant’, Reason=’Arbitration Mailbox is hosted on ‘DAG-1-DB0’, which is not a replicated database. ))
At C:Program FilesMicrosoftExchange ServerV14ScriptsStartDagServerMaintenance.ps1:216 char:16
+                 write-error <<<<  ($StartDagServerMaintenance_LocalizedStrings.res_0014 -f ( PrintCriticalMailboxResourcesOutput($criticalMailboxResources)),$shortServerName) -erroraction:stop
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Microsoft.PowerShell.Commands.WriteErrorCommand

 

If an administrator encounters this condition the following process can be utilized to place the DAG member into maintenance mode.  (In our example server DAG-1 in the DAG named “DAG” [pretty creative eh?] is the server we will be placing in maintenance mode)

 

1)  Execute a get-mailboxdatabasecopystatus * and verify that at least one other non-lagged copy of each replicated database is healthy.

 

[PS] C:>Get-MailboxDatabaseCopyStatus *

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                                              Length    Length                             State
—-                                          ——          ——— ———– ——————–   ————
DAG-1-DB0DAG-1                               Mounted         0         0                                  Healthy
DAG-DB0DAG-1                                 Mounted         0         0                                  Healthy
DAG-DB1DAG-1                                 Healthy         0         0           7/13/2011 8:22:55 AM   Healthy
DAG-2-DB0DAG-2                               Mounted         0         0                                  Healthy
DAG-DB1DAG-2                                 Mounted         0         0                                  Healthy
DAG-DB0DAG-2                                 Healthy         0         4           7/13/2011 8:48:34 AM   Healthy
DAG-DB0DAG-3                                 Healthy         0         147         7/13/2011 8:48:34 AM   Healthy
DAG-DB1DAG-3                                 Healthy         0         140         7/13/2011 8:22:55 AM   Healthy
DAG-DB0DAG-4                                 Healthy         0         409         7/13/2011 8:48:34 AM   Healthy
DAG-DB1DAG-4                                 Healthy         0         307         7/13/2011 8:22:55 AM   Healthy
MBX-1-DB0MBX-1                               Mounted         0         0                                  Healthy
MBX-1-RDBMBX-1                               Mounted         0         0                                  Healthy

2)  Execute a move of all active database copies off the server.  This can be done with the command move-activemailboxdatabase –server <MaintenanceServer>  (Note:  No target server is specified which means the next best copy will be automatically selected for activation)

 

[PS] C:>Move-ActiveMailboxDatabase -Server DAG-1 -Confirm:$FALSE

Identity        ActiveServerAtS ActiveServerAtE Status     NumberOfLogsLost   RecoveryPoint MountStatus MountStatus
                tart            nd                                            Objective     AtMoveStart AtMoveEnd
——–        ————— ————— ——     —————-   ————- ———– ———–
DAG-1-DB1       dag-1           dag-1           Warning                                     Mounted     Mounted
DAG-1-DB0       dag-1           dag-1           Warning                                     Mounted     Mounted
DAG-DB0         dag-1           dag-2           Succeeded  0                  7/13/2011 8:5 Mounted     Mounted
                                                                              3:24 AM
WARNING: An Active Manager operation failed. Error: The database action failed. Error: You cannot perform a switchover
operation on database ‘DAG-1-DB1’ because the database is not configured for replication.. [Database: DAG-1-DB1,
Server: DAG-1.domain.com]
WARNING: An Active Manager operation failed. Error: The database action failed. Error: You cannot perform a switchover
operation on database ‘DAG-1-DB0’ because the database is not configured for replication.. [Database: DAG-1-DB0,
Server: DAG-1.domain.com]

3)  Move the cluster core resources to another node within the DAG.  This can be accomplished using the command cluster.exe <DAGFQDN> group “Cluster Group” /moveto:<NODE>

 

[PS] C:>cluster DAG.domain.com group "Cluster Group" /moveto:DAG-2

Moving resource group ‘Cluster Group’…

Group                Node            Status
——————– ————— ——
Cluster Group        DAG-2           Online

4) Pause the node within the cluster.  This can be done utilizing the command cluster.exe <DAGFQDN> node <NODENAME> /pause

 

[PS] C:>cluster DAG.domain.com node DAG-1 /pause

Pausing node ‘DAG-1’…

Node           Node ID Status
————– ——- ———————
DAG-1                1 Paused

5) Set the DatabaseCopyAutoActivationPolicy of the server to BLOCKED.  This can be done using the command set-mailboxserver –identity <DAGMember> –databasecopyautoactivationpolicy:BLOCKED

 

[PS] C:>Set-MailboxServer -Identity DAG-1 -DatabaseCopyAutoActivationPolicy:BLOCKED

6) Suspend all individual copies for activation.  This can be done using the command get-mailboxdatabasecopystatus *<DAGMember> | suspend-mailboxdatabasecopy –activationOnly:$TRUE

 

[PS] C:>Get-MailboxDatabaseCopyStatus *DAG-1 | Suspend-MailboxDatabaseCopy -ActivationOnly:$TRUE
Database "DAG-1-DB0DAG-1" has only one copy. This task is supported only for databases that have more than one copy.
    + CategoryInfo          : InvalidOperation: (DAG-1-DB0:ADObjectId) [Suspend-MailboxDatabaseCopy], InvalidOperation
   Exception
    + FullyQualifiedErrorId : 7325D1AB,Microsoft.Exchange.Management.SystemConfigurationTasks.SuspendDatabaseCopy

Confirm
Are you sure you want to perform this action?
Suspending activation of mailbox database copy "DAG-DB0" on server "DAG-1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): a

 

At this time it should be safe for the administrator to perform DAG server maintenance.  When the maintenance is complete the script StopDagServerMaintenance.ps1 can be utilized to take the DAG member out of maintenance mode.

When implementing an Exchange 2010 Database Availability Group it may become necessary to perform a database seeding operation.  The operation is typically performed as part of adding a mailbox database copy to a DAG member but may also be performed to recover from database divergence.

Seeding is most often performed using the update-mailboxdatabasecopy command.  During seeding, the target replication service sends a seeding request to the source replication service on the DAG replication port (64327 by default).  The source replication service then initiates a local ESE streaming backup session to the Information Store service.  Pages are read from the source database by the source replication service and transmitted to the target replication service.  The target replication service then writes the pages to the target database.  There are sometimes where this process fails or for various reasons cannot be utilized.  This means an alternate way of seeding the database is needed.

One method is to perform a manual offline seeding.  In this operation, the source database is dismounted, verified, to be in a clean shutdown state, and then manually copied offline to the target.  This can obviously be inconvenient, since the source database has to be down while the copy procedure is being performed.

Another method is to use a VSS backup of the database to seed the database copy.  You can use VSS to backup the database, and VSS to restore the database.  (There are no longer streaming backups for Exchange 2010).

When using an Exchange-aware VSS application, there are typically four destinations for a restore (note, your backup software may not enable all the options):

1.  Original mailbox database.

2.  Alternate mailbox database.

3.  Recovery mailbox database.

4.  File system.

To use the VSS backup and restore method, you would choose to restore to the file system.

The following steps outline a high level process on how to utilize a VSS backup and restore to file system to complete an online offline database seed operation.

 

======================================

The first step is to enable replication for the mailbox database.  This step is accomplished by utilizing the add-mailboxdatabasecopy command with the –seedingPostponed parameter.  This command will add the copy and inform all replication services the copy is present.  Log truncation will also be suspended since copy status is not healthy for the database.  If –seedingPostponed is not specified the database seeding operation will automatically be performed.

add-mailboxdatabasecopy –identity <DBNAME> –mailboxServer <DAGMember> –seedingPostponed:$TRUE

 

[PS] C:Windowssystem32>Add-MailboxDatabaseCopy -Identity DAG-DB3 -MailboxServer MBX-3 -SeedingPostponed:$TRUE
WARNING: Replication is suspended for database copy ‘DAG-DB3’ because the database copy needs to be seeded.

 

If you have already added a mailbox database copy for the database proceed to the second step.

======================================

The second step is to ensure that the mailbox database copy is in a suspended state.  Mailbox database copies can be suspended in bulk or one at a time.  The following is an example command to suspend a single mailbox database copy:

 

Suspend-mailboxdatabasecopy –identity <DatabaseServer>

 

[PS] C:Windowssystem32>Suspend-MailboxDatabaseCopy -Identity DAG-DB3MBX-3

Confirm
Are you sure you want to perform this action?
Suspending mailbox database copy "DAG-DB3" on server "MBX-3".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): a

 

Prior to proceeding the get-mailboxdatabasecopy status command should be utilized to verify a status of suspended.  The following is an example command to verify copy status for a single database copy:

 

Get-mailboxdatabasecopystatus –identity <DatabaseServer>

 

[PS] C:Windowssystem32>Get-MailboxDatabaseCopyStatus DAG-DB3MBX-3

Name            Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                                Length    Length                             State
—-            ——          ——— ———– ——————–   ————
DAG-DB3MBX-3   Suspended       16        191         11/29/2010 7:49:53 AM  Failed

======================================

The third step is to note the important paths that are necessary to complete the rest of these steps.  Specifically, we are interested in the mailbox database log path and database file path.  To get all the paths for the mailbox database on the source server, use the following command:

 

get-mailboxdatabase –identity <Database> | fl name,logFilePrefix,logFolderPath,edbFilePath

 

[PS] C:Windowssystem32>Get-MailboxDatabase -Identity DAG-DB3 | fl name,logFilePrefix,logFolderPath,edbFilePath

Name          : DAG-DB3
LogFilePrefix : E04
LogFolderPath : c:DAGDAG-DB3
EdbFilePath   : c:DAGDAG-DB3DAG-DB3.edb

 

======================================

The forth step is to verify that the source log file sequence is in order.  If the source log file sequence has been manually manipulated, and if any log gaps are present, this results in a failure of the seed operation.  This step ensures that log files are in sequence on the source machine.

To ensure that the log sequence on the source machine is in the correct order, perform the following operations:

1)  Open a command prompt and navigate to the log file directory of the mailbox database.  This path can be found from the output gathered in step 3 above.

2)  Run the following eseutil command:

eseutil /ml <LogFilePrefix>

The log file prefix can be found from the output gathered in step 3.

When you run this command it will scan every log file found in the source directory.  If any gaps or errors are identified, you cannot continue with these steps.  If the command completes and errors on the last log file in the series this is expected, as the EXX.log is currently open for writing and cannot be scanned.  The following is sample output that you should receive for a mailbox database that is online.

 

[PS] C:DAGDAG-DB3>eseutil /ml E04

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode…

Verifying log files…
     Base name: E04

      Log file: C:DAGDAG-DB3E04000000BF.log – OK
      Log file: C:DAGDAG-DB3E04000000C0.log – OK
      Log file: C:DAGDAG-DB3E04000000C1.log – OK
      Log file: C:DAGDAG-DB3E04000000C2.log – OK
      Log file: C:DAGDAG-DB3E04000000C3.log – OK
      Log file: C:DAGDAG-DB3E04000000C4.log – OK
      Log file: C:DAGDAG-DB3E04000000C5.log – OK
      Log file: C:DAGDAG-DB3E04000000C6.log – OK
      Log file: C:DAGDAG-DB3E04000000C7.log – OK
      Log file: C:DAGDAG-DB3E04000000C8.log – OK
      Log file: C:DAGDAG-DB3E04000000C9.log – OK
      Log file: C:DAGDAG-DB3E04000000CA.log – OK
      Log file: C:DAGDAG-DB3E04000000CB.log – OK
      Log file: C:DAGDAG-DB3E04000000CC.log – OK
      Log file: C:DAGDAG-DB3E04000000CD.log – OK
      Log file: C:DAGDAG-DB3E04000000CE.log – OK
      Log file: C:DAGDAG-DB3E04000000CF.log – OK
      Log file: C:DAGDAG-DB3E04.log
                ERROR: Cannot open log file (C:DAGDAG-DB3E04.log). Error -1032.

Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use) after
1.469 seconds.

 

======================================

The fifth step is to perform a VSS backup of the database.  Please consult with your backup vendor to ensure that a successful FULL backup is performed.  Please also make sure that a consistency check of the backup is performed.

======================================

The sixth step is to restore the VSS backup.  When you perform the restore, you should select the option to restore to file system (it may be necessary to consult your backup vendor).  This may require that you restore to the file system of the Exchange Server, so it may be necessary to ensure that sufficient free space exists on a volume on the Exchange Server where the restore will be performed.

If multiple databases are being restored I recommend that databases be restored individually.

Please ensure that no recovery operations are performed on the database (options like roll forward recovery / replay logs / etc should be avoided).

At this point we now have the EDB file on the file system and we will use it for the seeding operation.

In our example we’ll restore to c:RESTORE.

======================================

The seventh step is to ensure that the target paths are ready to have the database moved in place.  The paths referenced in this step can be obtained from step 3.

In this example we will ensure the path c:DAGDAG-DB3 is empty on the target server.

 

 

If the paths already existed they are now ready to have the restored database moved to them.

If the paths do not exist they should be manually created.  If you are using nested folders you need to create the entire directory structure.

======================================

The eighth step is to move the restored database to the target directory.  This can be accomplished in a few different ways, but I will make a recommendation below.

From the source server map to the drive$ share of the target.  For example, I would map the Y drive to \MBX-3C$DAGDAG-DB3 using our example.

net use y: <path>

 

C:>net use y: \MBX-3c$DAGDAG-DB3
The command completed successfully.

 

On the source server open a command prompt to the directory where the data was restored.  In this example c:Restore

Use eseutil to copy the database from the source directory to the target directory.  A sample command:

eseutil /y <Source.EDB> /d <Target.EDB> Here is the output expected from the command using our example:

 

C:Restore>eseutil /y DAG-DB3.edb /d y:DAG-DB3.edb

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating COPY FILE mode…
     Source File: DAG-DB3.edb
Destination File: y:DAG-DB3.edb

                      Copy Progress (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
          ……………………………………………

        Total bytes read                = 0x18810000 (411107328) (392 MB)
        Total bytes written             = 0x18810000 (411107328) (392 MB)

Operation completed successfully in 29.16 seconds.

 

At this point the copy has been seeded on the target server. 

In place of the network administrators may consider a portable storage device for the transportation of the database to the target server.  (Note:  In this case eseutil /y would be used to copy the data to the portable storage and from the portable storage to the target server).

======================================

The ninth step is to verify the health of the copied database.  We need to ensure that the database was not corrupted as a part of the copy process.

On the target server open a command prompt and navigate to the location of the database file.  In our example this is c:DAGDAG-DB3.

Use the eseutil /k to perform a checksum of the database:

eseutil /k <Database.edb>

The following output will be observed when the command completes:

 

C:DAGDAG-DB3>eseutil /k DAG-DB3.edb

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 14.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating CHECKSUM mode…
        Database: DAG-DB3.edb
  Temp. Database: TEMPCHKSUM4804.EDB

File: DAG-DB3.edb

                     Checksum Status (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |—-|—-|—-|—-|—-|—-|—-|—-|—-|—-|
          ……………………………………………

12546 pages seen
0 bad checksums
0 correctable checksums
5571 uninitialized pages
0 wrong page numbers
0x62c97 highest dbtime (pgno 0x7c)

6273 reads performed
392 MB read
6 seconds taken
65 MB/second
5895209 milliseconds used
939 milliseconds per read
3437 milliseconds for the slowest read
93 milliseconds for the fastest read

Operation completed successfully in 6.672 seconds.

 

We are interested in ensuring that there are 0 bad checksums.

======================================

The last step in the process is to resume the mailbox database copy.

The following command can be used to resume the mailbox database copy:

Resume-MailboDatabaseCopy –identity <DatabaseServer>

 

[PS] C:>Resume-MailboxDatabaseCopy DAG-DB3MBX-3

 

Post a resume the following events can be noted in the application log on the target server.

 

Log Name:      Application
Source:        MSExchange Search Indexer
Date:          12/6/2010 5:30:16 AM
Event ID:      109
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
Exchange Search Indexer has created a new search index and will perform a full crawl for the Mailbox Database  DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225). Reason for full crawl: Catalog doesn’t exist.

Log Name:      Application
Source:        MSExchangeRepl
Date:          12/6/2010 5:30:17 AM
Event ID:      2114
Task Category: Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The replication instance for database DAG-DB3 has started copying log files. The first log file copied was generation 191.

Log Name:      Application
Source:        MSExchange Search Indexer
Date:          12/6/2010 5:30:18 AM
Event ID:      108
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
Exchange Search Indexer has enabled indexing for the Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225).

Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          12/6/2010 5:30:27 AM
Event ID:      1000
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
Attempting to start the Information Store "DAG-DB3".

Log Name:      Application
Source:        MSExchangeRepl
Date:          12/6/2010 5:30:37 AM
Event ID:      2157
Task Category: Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
The replication instance for database DAG-DB3 has copied and replayed multiple logs.

Log Name:      Application
Source:        MSExchange Search Indexer
Date:          12/6/2010 5:32:56 AM
Event ID:      110
Task Category: General
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-3.exchange.msft
Description:
Exchange Search Indexer completed a full crawl (indexing) of Mailbox Database DAG-DB3 (GUID = 124ba5d0-b20e-4297-8ac7-e1613dc86225).

The administrator can monitor the post seeding activity using the get-mailboxdatabasecopystatus <DatabaseServer> command. 

In this example we will use get-mailboxdatabasecopystatus DAG-DB3MBX-3. 

Initially the administrator will note a STATUS of RESYNCHRONIZING and a CONTENTINDEXSTATE of CRAWLING.

 

[PS] C:>Get-MailboxDatabaseCopyStatus DAG-DB3MBX-3

Name           Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                               Length    Length                             State
—-           ——          ——— ———– ——————–   ————
DAG-DB3MBX-3  Resynchronizing 20        0           12/5/2010 8:58:13 PM   Crawling

 

After all delta log files have copied to the target and replay has begun, the administrator will note a STATUS of HEALTHY and a CONTENTINDEXSTATE of CRAWLING.

 

[PS] C:>Get-MailboxDatabaseCopyStatus DAG-DB3MBX-3

Name           Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                               Length    Length                             State
—-           ——          ——— ———– ——————–   ————
DAG-DB3MBX-3  Healthy         0         50          12/6/2010 5:27:17 AM   Crawling

Due to the online offline database seed the content index files were not copied from the target server and were not restored from backup.  Therefore the search service on the target will initialize a new content index for this store and being indexing from the source.  If the database is large content indexing could be in a CRAWLING state for some time.

After the content index has been successfully built the administrator will note a STATUS of HEALTHY and a CONTENTINDEXSTATE of HEALTHY.

 

[PS] C:>Get-MailboxDatabaseCopyStatus DAG-DB3MBX-3

Name           Status          CopyQueue ReplayQueue LastInspectedLogTime   ContentIndex
                               Length    Length                             State
—-           ——          ——— ———– ——————–   ————
DAG-DB3MBX-3  Healthy         0         0           12/6/2010 5:27:17 AM   Healthy

 

At this point the database seeding operations are now 100% complete and the database and fully participate in DAG functions.

======================================

In Exchange 2003 and revisions prior the most common backup taken of an Exchange server leveraged the ESE Online Streaming APIs.  The use of ESE Online Streaming APIs carried forward into Exchange 2007 as customers migrated backup solutions to utilize the more common Volume Shadow Services method of backup for Exchange.  Exchange 2010 has fully deprecated the ESE Online Streaming APIs in favor of Volume Shadow Services.

A ESE Online Streaming backup occurs through three methods:

1)  Local shared memory.

2)  Remote TCP

3)  Remote RPC

Customers utilizing Windows 2003 / Exchange 2003 and Windows 2003 / Exchange 2007 can fully utilize any of the backup methods supported by NTBackup or a third party backup vendor.  Customers utilizing Windows 2008 / Exchange 2007 will not be able to take advantage of either Remote TCP or Remote RPC backups.  In other words, remote backups utilizing the online streaming API where the operating system is Windows 2008, regardless of what operating system the backup session initiates on or from, are explicitly not supported. 

The following are references to the support statements regarding remote streaming backups and Windows 2008:

 

http://msexchangeteam.com/archive/2008/03/05/448338.aspx

http://technet.microsoft.com/en-us/library/aa998870(EXCHG.80).aspx

Note:

Remote streaming backups or restores are not supported on or from Windows Server 2008.

 

Questions have recently been raised – how can I determine if my backup application is attempting to utilize a remote streaming backup against Windows 2008?

By default administrators attempting to utilize NTBackup from a Windows 2003 server or other workstation supporting NTBackup will always utilize a remote backup interface to the Exchange Server installation on Windows 2008.  The most common remote backup interface is TCP with failback to RPC should TCP not function.  For third parties this is dependent on the product installed.

The type of backup that is being performed will cause an event to be logged in the application log.  The following are sample events that occur when a REMOTE backup operation is being performed against an Exchange installation:

 

Time: 11/2/2009 12:26:02 PM
ID: 908
Level: Information
Provider: ESE BACKUP
Machine: <SERVER>
Message: Information Store (4816) Backup data transfer method is TCP/IP.

 

Event ID : 909
Raw Event ID : 909
Record Nr. : 277957
Category : Backup
Source : ESE BACKUP
Type : Information
Generated : 1/7/2009 12:42:04 AM
Written : 1/7/2009 12:42:04 AM
Machine : <SERVER>
Message : Information Store (4908) Backup data transfer method is RPC.

If either of these events are logged on the Windows 2008 server the type of backup that is being performed is NOT SUPPORT (regardless of the overall success or failure of the backup).  Unfortunately there are no supported workarounds to restore this functionality.

 

For customers utilizing Windows 2008 and desiring to retain an ESE Online Streaming API backup the only option for Exchange 2007 is the use of a third party product.  By default Windows 2008 does not include a backup technology capable of leveraging the ESE Online Streaming API.  Later revisions of Exchange 2007 introduced extensions to Windows Server Backup allowing Volume Shadow Services backups of Exchange 2007 to occur.  It is also not supported to port NTBackup from Windows 2003 to Windows 2008 for the purposes of Exchange ESE Online Streaming API backups.

Third party backup vendors leveraging the ESE Online Streaming API are able to do so by implementing a Local Shared Memory backup.  This is most commonly archived by installing a backup agent on the Windows 2008 server.  In these instances the backup server contacts the local backup agent.  The local backup agent establishes a local ESE Online Streaming API backup session.  The data is then read from the Information Store by the backup agent, where it is subsequently transmitted to the backup server. 

When a local shared memory backup is occurring the following event may be noted in the application log:

 

Event ID : 907
Raw Event ID : 907
Record Nr. : 24708
Category : Backup
Source : ESE BACKUP
Type : Information
Generated : 8/29/2007 12:20:36 AM
Written : 8/29/2007 12:20:36 AM
Machine : <SERVER>
Message : Information Store (3324) Backup data transfer method is shared memory (64kb).

 

To try to help clarify here is a sample chart that may assist:

Backup Initiator	Exchange Server	Supported
Online Streaming – Remote	Windows 2003 – All implementations of Exchange 2003 and Exchange 2007	Supported
Online Streaming – Local	Windows 2003 – All implementations of Exchange 2003 and Exchange 2007	Supported
Online Streaming – Remote	Windows 2008 – All implementations of Exchange 2007	NOT Supported
Online Streaming – Local (achievable only when using third party product)	Windows 2008 – All implementations of Exchange 2007	Supported

 

Customer are encouraged to evaluate backup methods that leverage the VSS Exchange aware backup methods for Exchange 2007 and Exchange 2010.

When using Windows Server Backup to backup Exchange 2007 and Exchange 2010 there are essentially four processes that must occur in order for the backup to be successful:

 

1)  Volume Shadow copy is made of the Exchange data on the host where the backup is initiated.

2)  All Exchange log files and database files in the backup set must have a consistency check performed.

3)  The data on the drives is written to the desired backup media.

4)  Backup complete is performed allowing for log truncation activities (if applicable).

 

Step #1:  Create shadow copies

 

The first part of all Exchange backups using Windows Server Backup is the creation of the shadow copy.  An administrator should expect the shadow copy process to complete quickly.  The previous screen shot provides an example of the Windows Server Backup status.  The following events may also be noted in the application log.

 

Time:     11/28/2010 12:29:47 PM
ID:       9606
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947) has prepared for backup successfully.

Time:     11/28/2010 12:29:49 PM
ID:       2005
Level:    Information
Source: ESE
Machine:  Server1-A.domain.com
Message:  Information Store (5340) Shadow copy instance 2 starting. This will be a Full shadow copy.

For more information, click http://www.microsoft.com/contentredirect.asp.

Time:     11/28/2010 12:29:49 PM
ID:       9811
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 2) has successfully prepared the database engine for a full or copy backup of database ‘DATABASE’.

Time:     11/28/2010 12:29:50 PM
ID:       2001
Level:    Information
Source: ESE
Machine:  Server1-A.domain.com
Message:  Information Store (5340) Shadow copy instance 2 freeze started.

For more information, click http://www.microsoft.com/contentredirect.asp.

Time:     11/28/2010 12:29:50 PM
ID:       2001
Level:    Information
Source: ESE
Machine:  Server1-A.domain.com
Message:  Information Store (5340) DATABASE: Shadow copy instance 2 freeze started.

For more information, click http://www.microsoft.com/contentredirect.asp.

Time:     11/28/2010 12:29:50 PM
ID:       9610
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947:2) has frozen the database(s) successfully.

Time:     11/28/2010 12:29:52 PM
ID:       2003
Level:    Information
Source: ESE
Machine:  Server1-A.domain.com
Message:  Information Store (5340) Shadow copy instance 2 freeze ended.

For more information, click http://www.microsoft.com/contentredirect.asp.

Time:     11/28/2010 12:29:52 PM
ID:       9612
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947:2) has thawed the database(s) successfully.

Time:     11/28/2010 12:29:54 PM
ID:       9622
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947:2) has processed the post-snapshot event successfully.

 

Step #2:  Running consistency check

 

The second part of the backup is running consistency check.  The consistency check will verify the integrity of each log file and each database that is a part of the backup set.  Should a corrupted log file <or> database be found, the backup job will be cancelled.  The consistency check is not optional and must occur in order for the backup to continue. 

 

The length of time that a consistency check takes is dependent on two main factors:

1)  The number of log files in the backup set.

2)  The size of the database files.

 

Recently we have seen several cases where it appears that consistency check never completes or in some instances takes multiple hours / days to complete.  In most cases there was a common theme – the number of log files to be backed up.  In many of these cases they were the first backup taken after a migration, when log file creation is high and the number of logs in the backup set is large.   (For example, in one case reviewed there were over 50 gigs of log files in the backup set along with a 750 gig mailbox store).

 

When consistency check fails to complete in the desired time there are very limited options.  The only option that is generally effective is lowering the size of the backup set.  For example, this may require manually removing log files (it would be advisable to contact support for assistance) or lower the size of the database file.  For customers using a Database Availability Group, you may also be able to distribute your databases among several DAG members, and backup those individual members.  In the DAG example the backup set size is not changing but only the number of active databases require consistency checking, which lowers the size of the backup set requiring consistency check.

 

Step #3:  Writing data to backup media

 

At this stage the shadow copies and consistency check has completed.  The data in the backup set is now being streamed to the user specified location. 

 

The amount of time for data transfer is both dependent on the overall size of the backup set as well as the destination chosen.  The backup cannot be fully completed until all data in the backup set has been streamed to media (this includes non-Exchange related data such as system state / bare metal recovery).

 

Step #4:  Backup complete

 

At this stage the shadow copies, consistency check, and data transfer to media has completed.  At this time the Exchange writer can be informed that the backup completed successfully.  It is also at this time that if a full backup was performed that log truncation can occur for logs that are eligible for truncation.  This is a very important step, if backup complete is not called by the application future backups are subject to fail with an error that a backup is already in progress.  (This may necessitate restarting the Information Store service in order to clear this condition).  Please note that log truncation no longer occurs as a part of the backup process, therefore it is not necessary for logs to truncate in order for the backup to be completed successfully.  It may be several minutes until the log file truncation is actually processed on the hosts.

 

The following events may be found in the application log regarding the Exchange backup.

 

Index:    2
Time:     11/28/2010 12:50:06 PM
ID:       224
Level:    Information
Source: ESE
Machine:  Server1-A.domain.com
Message:  Information Store (5340) DATABASE: Deleting log files d:LogsDATABASEE020000201E.log to d:LogsDATABASEE0200002046.log.

For more information, click http://www.microsoft.com/contentredirect.asp.

Index:    3
Time:     11/28/2010 12:50:06 PM
ID:       9780
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947:2) has successfully completed the full or incremental backup of storage group ‘DATABASE’.

The database engine has also successfully executed log file truncation procedures for this storage group. (Note that this may or may not have resulted in the actual truncation of log files, depending on whether any log files existed that were candidates for truncation.)

Index:    12
Time:     11/28/2010 12:50:06 PM
ID:       2006
Level:    Information
Source: ESE
Machine:  Server1-A.domain.com
Message:  Information Store (5340) Shadow copy instance 2 completed successfully.

For more information, click http://www.microsoft.com/contentredirect.asp.

Index:    13
Time:     11/28/2010 12:50:06 PM
ID:       9616
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947:2) has processed the backup completion event successfully.

Index:    14
Time:     11/28/2010 12:50:07 PM
ID:       9648
Level:    Information
Source: MSExchangeIS
Machine:  Server1-A.domain.com
Message:  Exchange VSS Writer (instance 3d2f77ec-302d-42d4-9eac-e38082398947:2) has processed the backup shutdown event successfully.

Over the last few days I’ve received questions from several people inquiring if it is necessary and / or a best practice to perform a backup after a restoration of a database has occurred.

In general I do not think there is a best practice / guidance surrounding this question – in most cases the answer is it depends.

When I work a support case that necessitates the restoration of a database I will generally advise customers to perform a full backup once the database is in service and when the backup will not interfere with production hours. 

For some customers the nature of the restoration makes this a higher priority – for example:

 

1)  There were several days / hours of logs to play through (it would be nice to have a more current anchor point after recovery). 

2)  I had to resort to a hard repair of the database.  (This process essentially invalidates all previous backups such that the next full backup would become the anchor point for future recovery).

3)  Overall I’m not confident the issue that caused the need for restoration was actually addressed (it would be nice to have a more current point of recovery in anticipation that we’ll be doing it again soon).

 

There are also another set of customers where capturing a full backup is advised – those customers that are using an incremental backup strategy.  After a database is restored the previous full backup information is cleared from the header of the database – this would indicate that further incremental backups will fail with an error message that a full backup needs to be performed.  Therefore, customers that use a full / incremental backup strategy it would be advisable to perform a full backup prior to the next scheduled incremental so that backup job failures do not result.

(You can do the get-mailboxdatabase –status command to see the times of the last previous full and last previous incremental backups).

 

I think it goes without saying – you really cannot go wrong with ensuring that you have a current backup after any type of recovery operation.