A common question that has come up through support is how can administrators determine if forwarding has been enabled on an account.  There are three main methods to enable forwarding on a mailbox:

• The client creates a rule for either redirect or forwarding using Outlook.
• The client or administrator sets the forwardingSMTPAddress property of the users mailbox.
• The administrator sets the forwardingAddress property of the users mailbox.

As you can see two of the three options are controlled by the end users – and therefore may occur without the administrator or the organization being aware.

Using powershell the administrator can begin to determine which of these options are enabled on users mailboxes. 

The process begins by establishing a remote powershell session to Office 365.

$cred=get-credential

$session=new-pssession –configuration Microsoft.Exchange –connectionURI https://ps.outlook.com/powershell –authentication BASIC –allowRedirection:$TRUE –credential $cred

import-psSession $session

The mailboxes in the organization are gathered into a variable that will be used for further analysis.

$mailboxes=get-mailbox –resultSize unlimited

The mailboxes are then evaluated for inbox rules that have either redirect or forwarding.  Using an inbox rule the client can configure a forward flag, a forwarding attachment flag, or a redirect flag.  The rules that exist in the organization are gathered into a variable that will be used for further analysis.

$rules = $mailboxes | foreach { get-inboxRule –mailbox $_.alias }

The rules found are then processed to determine whether a forward or redirect flag is enabled.$rules

$rules | where { ( $_.forwardAsAttachmentTo –ne $NULL  ) –or ( $_.forwardTo –ne $NULL ) –or ( $_.redirectTo –ne $NULL ) } | ft name,identity,ruleidentity

This produces a list of all mailboxruleIDs that exist in the organization where the forwarding or redirect flags are enabled.

Name                           Identity                                          RuleIdentity
—-                           ——–                                          ————
ForwardToExternal              MAILBOX113158606040511545345             13158606040511545345
Redirect                       MAILBOX22296058644233453570               2296058644233453570
ForwardAsAttachment            MAILBOX22368116238271381506               2368116238271381506

The specifics of the rule can be determined with the get-inboxRule command.

Get-InboxRule –identity “Mailbox22296058644233453570” | fl name,forwardTo,forwardAsAttchmentTo,redirectTo

Name                  : ForwardAsAttachment
ForwardTo             :
ForwardAsAttachmentTo : {"RECIPIENT”}
RedirectTo            :

This covers the mailboxes that have rules created for forwarding or redirection.

The forwardingSMTPAddress flag can be set by the end users through Outlook Web Access or by the administrator through a powershell command. 

An example of Outlook Web Access:

An example of powershell:

Set-Mailbox Administrator -ForwardingSmtpAddress:tim@contoso.com

The administrator can use the same method for inbox rules to investigate this attribute. 

$mailboxes | where { $_.forwardingSMTPAddress –ne $NULL }

This command generates a list of mailboxes where the forwarding SMTP address has been set.

Name                      Alias                ServerName       ProhibitSendQuota
—-                      —–                ———-       —————–
MailboxName               Alias                bn1pr06mb101     49.5 GB (53,150,220,288 bytes)
MailboxName2              Alias2               blupr06mb417     49.5 GB (53,150,220,288 bytes)

The last method of forwarding is the forwardingAddress flag.  The forwarding address flag is set by the administrator. 

Through the Exchange Control Panel the administrator sets this flag by selecting the mailbox –> mailbox features –> delivery options. 

Through powershell the administrator executes the set-mailbox command.

set-mailbox –identity MAILBOX –forwardingAddress Contact

Using the same variable previously created the administrator can also discover mailboxes where forwardingAddress is set.

$mailboxes | where { $_. forwardTo –ne $NULL }

This command displays a list of mailboxes where the forwardTo flag is set.

Name         Alias      ServerName       ProhibitSendQuota
—-         —–      ———-       —————–
Mailbox      Alias      co1pr06mb175     49.5 GB (53,150,220,288 bytes)

By using powershell administrators can discover mailboxes that utilize a forwarding action.

When migrating mailboxes to Office 365 administrators have the opportunity to have the move batch immediately completed or manually complete the batch at a later time.  The move to Office 365 for the majority of mailboxes is an online move – allowing the end user full access to their mailbox while the move process occurs.

An online move to Office 365 occurs in multiple stages:

• Enumerating and synchronization of initial folders.
• Transfer and subsequent completion of the initial synchronization.
  • If the move is not finalized incremental synchronizations will occur daily.
• Finalization of the move by detecting delta differences, locking the source and target mailboxes, migrating final changes, and completing required recipient object updates.
  • This process may occur automatically if the batch is selected to complete or manually when the administrator completes the batch.

The time to completion of the finalization of the move is largely dependent on the number of changes that have occurred between the initial seeding and incremental synchronization.  Some factors that may contribute to this:

• How much time elapsed between the initial synchronization and when the batch was manually completed?
• How long did it take to move the original mailbox data before finalization automatically occurred?
• What type of usage does the mailbox experience while the move is in progress?
• Were there other processes responsible for changing large numbers of messages?
  • For example, retention policies / archiving / etc.
• The latency between the Office 365 datacenter and the on premises endpoint used for migration.

In Office 365 migration batches are handled by the migration service.  This is the service responsible for creating the underlying move requests and monitoring their status. 

Let’s take a look at an example of how this affects a move.  A migration batch is initiated for a single mailbox.  This results in the associated move request being created by the migration service.  The move request is picked up by the appropriate mailbox replication service.

​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

2/18/2015 10:59:08 PM [CO1PR06MB111] '' created move request.
2/18/2015 10:59:08 PM [CO1PR06MB111] '' allowed a large amount of data loss when moving the mailbox (50 bad items, 50 large items).
2/18/2015 10:59:23 PM [BLUPR06MB580] The Microsoft Exchange Mailbox Replication service 'BLUPR06MB580.namprd06.prod.outlook.com' (15.1.87.18 caps:1FFF) is examining the request.

Connections are created between the target and source mailboxes.

2/18/2015 10:59:23 PM [BLUPR06MB580] Connected to target mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)', database 'NAMPR06DG037-db096', Mailbox server 'BLUPR06MB580.namprd06.prod.outlook.com' Version 15.1 (Build 87.0).
2/18/2015 10:59:23 PM [BLUPR06MB580] Connected to target mailbox 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)', database 'NAMPR06DG037-db096', Mailbox server 'BLUPR06MB580.namprd06.prod.outlook.com' Version 15.1 (Build 87.0).
2/18/2015 10:59:26 PM [BLUPR06MB580] Connected to source mailbox 'TENANT.onmicrosoft.comSOURCE-PRIMARY-GUID (Primary)', database 'SOURCEDB', Mailbox server 'SERVER.DOMAIN.com' Version 14.3 (Build 123.0), proxy server 'HYBRID.DOMAIN.com' 14.3.123.2 caps:05FFFF.
2/18/2015 10:59:29 PM [BLUPR06MB580] Connected to source mailbox 'TENANT.onmicrosoft.comSOURCE-ARCHIVE-GUID (Archive)', database 'SOURCEDB', Mailbox server 'SERVER.DOMAIN.com' Version 14.3 (Build 123.0), proxy server 'HYBRID.DOMAIN.com' 14.3.123.2 caps:05FFFF.

The folder hierarchy for the remote mailbox is completed.  At this stage the mailbox hierarchy is synchronized from the source mailbox to the target mailbox.

2/18/2015 10:59:46 PM [BLUPR06MB580] Initializing folder hierarchy from mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 239 folders total.
2/18/2015 10:59:47 PM [BLUPR06MB580] Folder creation progress: 0 folders created in mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)'.
2/18/2015 11:04:48 PM [BLUPR06MB580] Folder creation progress: 103 folders created in mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)'.
2/18/2015 11:09:51 PM [BLUPR06MB580] Folder creation progress: 218 folders created in mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)'.
2/18/2015 11:10:37 PM [BLUPR06MB580] Folder hierarchy initialized for mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 238 folders created.

At this stage the initial synchronization of data is started.

2/18/2015 11:13:12 PM [BLUPR06MB580] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
2/18/2015 11:13:16 PM [BLUPR06MB580] Initial sync checkpoint progress: 0/239 folders processed. Currently processing mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)'.
2/18/2015 11:18:16 PM [BLUPR06MB580] Initial sync checkpoint progress: 185/239 folders processed. Currently processing mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)'.
2/18/2015 11:21:09 PM [BLUPR06MB580] Initial sync checkpoint completed: 288 folders processed.
2/18/2015 11:21:09 PM [BLUPR06MB580] Stage: LoadingMessages. Percent complete: 20.
2/18/2015 11:31:08 PM [BLUPR06MB580] Messages have been enumerated successfully. 211371 items loaded. Total size: 17.26 GB (18,528,022,914 bytes).
2/18/2015 11:31:08 PM [BLUPR06MB580] Stage: CopyingMessages. Percent complete: 25.
2/18/2015 11:31:08 PM [BLUPR06MB580] Copy progress: 0/211371 messages, 0 B (0 bytes)/17.26 GB (18,528,022,914 bytes), 7/63 folders completed.
2/18/2015 11:36:27 PM [BLUPR06MB580] Stage: CopyingMessages. Percent complete: 25.
2/18/2015 11:36:27 PM [BLUPR06MB580] Copy progress: 911/211371 messages, 10.6 MB (11,117,242 bytes)/17.26 GB (18,528,022,914 bytes), 7/63 folders completed.
2/18/2015 11:45:16 PM [BLUPR06MB580] Stage: CopyingMessages. Percent complete: 25.
2/18/2015 11:45:16 PM [BLUPR06MB580] Copy progress: 1183/211371 messages, 34.57 MB (36,247,647 bytes)/17.26 GB (18,528,022,914 bytes), 7/63 folders completed.
2/18/2015 11:52:26 PM [BLUPR06MB580] Stage: CopyingMessages. Percent complete: 25.

The synchronization of messages will continue until all messages from the initial synchronization state have been migrated.

2/24/2015 2:50:17 AM [BLUPR06MB580] Initial seeding completed, 213714 items copied, total size 17.26 GB (18,529,584,134 bytes).

When the initial seeding is completed the mailbox move status is at 95%.  If the administrator has selected to manually complete the batch the moves will remain until completed.  If the administrator selected the option to immediately finalize the move the incremental synchronization process will automatically start.

2/24/2015 2:50:17 AM [BLUPR06MB580] Initial seeding completed, 213714 items copied, total size 17.26 GB (18,529,584,134 bytes).
2/24/2015 2:50:17 AM [BLUPR06MB580] Stage: IncrementalSync. Percent complete: 95.

Once the incremental synchronization process has commenced we begin by enumerating all changes that have occurred since the initial synchronization.

2/24/2015 3:02:51 AM [BLUPR06MB580] Content changes reported for mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': New 0, Changed 2514, Deleted 112207, Read 0, Unread 0, Total 114721 .
2/24/2015 3:10:10 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 100/2514 changed items processed.
2/24/2015 3:16:13 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 300/2514 changed items processed.
2/24/2015 3:21:39 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 1400/2514 changed items processed.
2/24/2015 3:29:14 AM [BLUPR06MB580] Total content changes applied to mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': New 0, Changed 2514, Deleted 112209, Read 0, Unread 0, Skipped 0, Total 114723.

After determining the change set the changes are synchronized to the mailbox.

2/24/2015 3:10:10 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 100/2514 changed items processed.
2/24/2015 3:16:13 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 300/2514 changed items processed.
2/24/2015 3:21:39 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': 1400/2514 changed items processed.
2/24/2015 3:29:14 AM [BLUPR06MB580] Total content changes applied to mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': New 0, Changed 2514, Deleted 112209, Read 0, Unread 0, Skipped 0, Total 114723.

Additionally this mailbox had an archive mailbox which uses the same process above.  After synchronization of the primary mailbox the new items in the archive mailbox is performed.

2/24/2015 3:51:32 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 700/112156 changed items processed.
2/24/2015 3:58:10 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 1100/112156 changed items processed.
2/24/2015 4:03:11 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 1300/112156 changed items processed.
2/24/2015 4:12:59 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 1600/112156 changed items processed.
2/24/2015 4:20:40 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 1900/112156 changed items processed.
2/24/2015 4:28:09 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 2000/112156 changed items processed.

Approximately 5 hours later the move request is automatically suspended, we remain at 95% completed, and the move request enters a Completed with Error state.

2/24/2015 7:50:45 AM [CO1PR06MB111] '' suspended move request.
2/24/2015 7:55:23 AM [BLUPR06MB580] Applying changes to 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': 8400/112156 changed items processed.
2/24/2015 7:55:23 AM [BLUPR06MB580] Suspending job.
2/24/2015 7:55:23 AM [BLUPR06MB580] Stage: IncrementalSync. Percent complete: 95.
2/24/2015 7:55:26 AM [BLUPR06MB580] Relinquishing job.

In this particular case resuming the request will result in the same failure repeatedly. Why did this occur?  In this particular example you’ll note that there was a large degree of delta change between the initial mailbox seeding and the incremental sync. 

2/24/2015 3:29:14 AM [BLUPR06MB580] Total content changes applied to mailbox 'TENANT.onmicrosoft.comTARGET-PRIMARY-GUID (Primary)': New 0, Changed 2514, Deleted 112209, Read 0, Unread 0, Skipped 0, Total 114723.

2/24/2015 3:43:45 AM [BLUPR06MB580] Content changes reported for mailbox 'TENANT.onmicrosoft.comTARGET-ARCHIVE-GUID (Archive)': New 0, Changed 112156, Deleted 0, Read 0, Unread 0, Total 112156 .

Specifically a large number of new messages committed to the archive mailbox.  In this particular case there was a retention policy responsible for moving items from the primary mailbox to the archive mailbox.  Therefore the items in the archive mailbox were new, arrived after the original synchronization set was determined, and would need to be moved during incremental synchronization.   During the incremental synchronization the move request was timed out within the service and the job suspended.  The time out that is applied to incremental synchronization is subject to change. 

To correct this condition we suspended all retention activities against the mailbox.  This kept the delta change set to only those items generated by the end user and normal mail flow activity.  The migration batch was removed and a new migration batch completed.  In this instance both the migration of the primary and archive mailbox completed without any issue.

For many Office 365 customers directory synchronization is the primary method of populating their Office 365 tenant with users, groups, and contacts.  The directory synchronization process allows for the centralized management of these objects.

In the service customers taking advantage of Exchange Online through Office 365 actually have two directory representations.  Objects managed with directory synchronization are replicated to the Azure Active Directory.  This directory serves as the centralized repository for Office 365 objects.  The directory synchronization process is responsible for mapping on premises Active Directory attributes to the Azure Active Directory.  Each object within the on premises active directory has a wide array of attributes that are configurable on the object.  The directory synchronization service is configured to map these attributes to attributes within the Azure Active Directory. 

Attributes of objects within the on premises Active Directory are generally managed by either Active Directory Users and Computers or the Exchange Management tools.  In some special circumstances tools like ADSIEdit can be utilized to process attribute changes.  When an object is changed within the directory administrators can view the synchronization of those changes through the MIISClient on the Directory Synchronization server. 

In this example I have changed an object on premises by adding City. 

In the MIISConsole we can see the object change detected and subsequently the attribute updated in the Azure Active Directory.  The original CITY attribute was NULL and has now been populated with the value “Charlotte”.

Using the Azure Active Directory module we can see the results of this change with get-MSOLUser.

PS C:> Get-MsolUser -UserPrincipalName DevUser@domain.com | fl userprincipalname,city

UserPrincipalName : DevUser@domain.com
City              : Charlotte

The list of attributes that are synchronized from On-Premises to the Azure Active Directory is documented at this location:

http://support.microsoft.com/kb/2256198

http://social.technet.microsoft.com/wiki/contents/articles/19901.list-of-attributes-that-are-synced-by-the-windows-azure-active-directory-sync-tool.aspx

The second directory representation for the object is present within the Exchange Online directory.  When an object has been assigned an Exchange license, a service provisioning process is responsible for extending this object into the Exchange Online directory and subsequently providing a mailbox.  This provisioning process is also responsible to monitor for changes and ensure those changes are reflected within the Exchange Online object.  For example, if you look at the mailbox associated with our test user you will now see that city has been populated with a value of “Charlotte” where previously it was NULL.

PS C:> Get-Recipient -Identity DevUser@domain.com | fl name,city

Name : Dev User
City : Charlotte

The attributes that are replicated from the Azure Active Directory to the Exchange Online directory are not necessarily the same as those replicated from the On-Premises Active Directory to the Azure Active Directory.  Currently there is no list of attributes that replicate between Azure Active Directory and Exchange Online.

In most instances no issues arise from this difference in directory synchronization.  There are though some circumstances where this can cause an issue.   Let us explore an example. 

An administrator has decided that on-premises an address list would exist that displays all recipient objects where the Company attribute is populated with a given value.  The address list allows for you to specify that any recipient that contains company=X will be included in the address list.  This effectively allows the administrator to set company on users, distribution groups, and contacts and have them appear in the same address list.

The directory synchronization metaverse, by default, does not sync the company attribute on objects of type group.  Notice in the below example that company is not included.

Using the add attribute button Company is added as an attribute that is synchronized for groups.

By default company is not an attribute of a group when edited via Active Directory Users and computers or through the Exchange Management Tools. 

To achieve this result the administrator uses the attribute editor of Active Directory Users and Computers (or ADSIEdit) to set the company attribute on the distribution group.

When reviewing the directory synchronization logs we can observe the object is replicated with the attribute company populated.

After waiting a period of time it was noted that the same address list in Exchange Online did not contain the group objects.  This would be considered by design – Company is not an expected attribute of a distribution group.  Although the Azure Active Directory did in this case allow for the attribute to be successfully synchronized from on premises the Exchange Online provisioning process is not configured to detect this attribute and replicate it to Exchange online.

In this case the administrator needed to switch to an attribute that is replicated, for example custom attribute 1.  There should be no expectation that attributes that are not default on an object on-premises will be replicated into Exchange Online although they may be replicated to the Azure Active Directory. 

Customers may implement CRM applications that track customer interactions.  In some instances the CRM applications are configured to archive and record email transactions that are sent to and from customer service representatives. 

I recently worked with a customer that has their CRM application configured to monitor a centralized mailbox.  When a new organization was added to the CRM system, a proxy address would be added to this centralized mailbox for that customer.  For example, I might have a  centralized mailbox shared@company.com.  When a new customer is added to the CRM system, this mailbox would have an additional secondary proxy added of customer1@company.com.  When the customer service representatives communicate with this customer by email, the customer1@company.com alias is added to the message CC line.  This ensures that all messages to that customer and subsequent replies are retained in the centralized mailbox.  This also allows the CRM application, which monitors this centralized mailbox, to ingest the information into the CRM application.

The issue arises with how Outlook, Outlook Web Access, and transport handle secondary proxies.  When a secondary proxy is addresses in any of these products the proxy is immediately resolved to the user object where the proxy is applied.  Instead of retaining customer1@company.com on the CC line the address is changed to shared@company.com.  This results in exposing this mailbox / address in all correspondence to external recipients rather then the individual customer proxy address added to the object.

Let’s take a look. 

For this example I have created a mailbox CentralMailbox@domain.com.  This will serve as the mailbox that I would have subscribed for monitoring by my CRM application. 

PS C:> Get-Mailbox CentralMailbox

Name                      Alias                ServerName       ProhibitSendQuota
—-                      —–                ———-       —————–
CentralMailbox            CentralMailbox       by2pr06mb028     49.5 GB (53,150,220,288 bytes)

On this mailbox I have added a secondary proxy address of Customer1@domain.com. 

PS C:> $a=Get-Mailbox centralmailbox
PS C:> $a.emailAddresses
smtp:customer1@domain.com
SIP:CentralMailbox@tenant.onmicrosoft.com
SMTP:CentralMailbox@tenant.onmicrosoft.com

When addressing a message in OWA I manually utilize the customer1@domain.com email address. 

When the email is sent to the external recipient the reply to address is not the secondary address but rather the primary address of the mailbox where the secondary proxy is assigned. 

You can also see in the sent items that the address was resolved to the mailbox where it was assigned and not retained as an SMTP address.

How can this be resolved? 

One potential solution is to utilize customer mailboxes rather than secondary proxy addresses for customer communications.  To prevent having to configure the CRM application to monitor multiple mailboxes we would utilize the forwarding and delivery features in Exchange and Office 365.  Let’s take a look at an example.

In this instance a mailbox was created Customer2@domain.com

PS C:> Get-Mailbox Customer2

Name                      Alias                ServerName       ProhibitSendQuota
—-                      —–                ———-       —————–
Customer2                 Customer2            dm2pr0601mb0921  49.5 GB (53,150,220,288 bytes)

There are two properties of the mailbox that allow us to utilize this as a solution – ForwardingAddress and DeliverToMailboxAndForward. 

PS C:> Get-Mailbox Customer2 | fl ForwardingAddress,DelivertoMailboxandForward

ForwardingAddress          :
DeliverToMailboxAndForward : False

Using powershell I will set the ForwardingAddress to the centralized mailbox and DeliverToMailboxAndForward to FALSE.  By keeping DeliverToMailboxandForward false the message will not be delivered to the customer mailbox but only the central mailbox.  Should you desire to have it delivered to both location this would be set to TRUE.

PS C:> Set-Mailbox Customer2 -ForwardingAddress CentralMailbox -DeliverToMailboxAndForward:$FALSE

PS C:> Get-Mailbox Customer2 | fl ForwardingAddress,DelivertoMailboxandForward

ForwardingAddress          : CentralMailbox
DeliverToMailboxAndForward : False

When an email is addressed in OWA and the customer email address utilized it resolves to the mailbox that was created for the customer.

When the external customer receives the email the customer specific mailbox is retained – the centralized mailbox is not exposed.

When the external user performs a reply all the email continues to be addressed to the customer mailbox rather than the centralized mailbox.

When logging into the centralized mailbox both the original message and the reply are present.  The message retained the Customer2 email address even through they were forwarded to the centralized mailbox.

Utilizing a mailbox and mailbox forwarding allows us to address emails without exposing the central mailbox but still capture all correspondence in the central mailbox.  Not allowing delivery to the customer mailbox ensures that the message is stored in one location and the mailbox itself will not utilize any storage (although it will be assigned an Exchange database).

Another potential option utilizes distribution groups.  Let’s take a look.

Instead of creating a mailbox for each customer a distribution group is created. 

PS C:> Get-DistributionGroup Customer3

Name                          DisplayName                   GroupType                     PrimarySmtpAddress
—-                          ———–                   ———                     ——————
Customer3                     Customer3                     Universal                     Customer3@fortmillrescue.com

The central mailbox is added as a member to the distribution group.

PS C:> Add-DistributionGroupMember -Identity Customer3 -Member CentralMailbox

PS C:> Get-DistributionGroupMember Customer3

Name                                                        RecipientType
—-                                                        ————-
CentralMailbox                                              UserMailbox

In this example the only member of the distribution list is the central mailbox.

The distribution list must also be set to accept mail from all senders, specifically senders that are non-authenticated.  This is required with the assumption that email will be received from external senders that are not authenticated to the hosting message solution.

PS C:> Set-DistributionGroup Customer3 -RequireSenderAuthenticationEnabled:$FALSE

When addressing an email in OWA the customer email is added to the CC line.  The SMTP address resolves automatically to the distribution list we created.

When the message is received by the external recipient this email address is retained.  The centralized mailbox is not exposed to the external recipient.

As in the mailbox example subsequent replies are addressed to the distribution list which results in both the original message and the replies residing in the centralized mailbox for processing.

Internally those users utilizing a distribution group may be able to expand distribution list membership in the message exposing the central mailbox to both internal and external users.  Additional internal users can pull the properties of the distribution list in the Global Address List and view membership.  Utilizing a distribution list is another viable option to ensure emails are delivered to the central mailbox for processing while preserving the customer specific email address.

Customer may additionally want to hide the customer mailboxes or distribution lists from the global address list.  This does not alter the functionality documented here but would prevent the selection of these objects from the global address list.  Knowledge of the specific customer email addresses would be necessary in order to address emails as no automatic name resolution would occur.

*Special thanks to Mark Terry for the additional suggestion of utilizing distribution groups in this scenario.*

In Exchange 2010 and Exchange 2013 the process of mounting a database has changed since Exchange 2007.  In versions prior to Exchange 2010 the Information Store process on a server was solely responsible for tracking the creation of mailbox databases and performing mount and dismount operations.

In Exchange 2010 and newer the Active Manager process is now responsible for tracking the creation of mailbox databases and performing mount / dismount operations.  The Active Manager process works with the Information Store process to complete these requests.  Every server, both standalone and members of Database Availability Groups, have an Active Manager process.  This process runs within the Exchange Replication Service.

When administrators create a new mailbox database an entry is created in the database configuration container within Active Directory.  This will subsequently replicate to all other domain controllers.  Some administrators have noted that if you attempt to create a new database and immediately mount it – it fails with the following error:

[PS] C:>Mount-Database test
Failed to mount database "test". Error: An Active Manager operation failed. Error: Couldn't find the specified mailbox
database with GUID '1a5e5284-a2d3-4241-a31a-fa0023fdc0e4'. [Database: Test, Server: MBX-3.domain.com]
    + CategoryInfo          : InvalidOperation: (Test:ADObjectId) [Mount-Database], InvalidOperationException
    + FullyQualifiedErrorId : [Server=MBX-1,RequestId=8bcb44ca-f7cf-4594-81b9-1adb70b5cb95,TimeStamp=2/1/2015 3:47:30
   PM] [FailureCategory=Cmdlet-InvalidOperationException] 19FDD999,Microsoft.Exchange.Management.SystemConfigurationT
  asks.MountDatabase
    + PSComputerName        : mbx-1.domain.com

This error may occur more often when creating a database and checking the mount this database option (checked by default).

When a mount command is issued the command is referred to the Active Manager process on the server.  In the case of a Database Availability Group the command is referred to the Primary Active Manager and then passed to the Replication Service of the node owning the active mailbox database copy.  The Replication Service then send a notification to the Information Store process to perform the mount.  In this instance the Replication Service is aware of the presence of the database, but the Information Store process is not.  This ultimately returns an error to the Replication Service which is subsequently returned to the administrative endpoint where the mount was issued (PowerShell / ECP).

After waiting a few minutes the database can be successfully mounted.  The wait time may be longer depending on the amount of time it takes domain controllers to successfully replicate the database object.

[PS] C:>Mount-Database Test

[PS] C:>Get-MailboxDatabase Test -Status | fl name,*mounted*

Name            : Test
MountedOnServer : MBX-1.domain.com
Mounted         : True

The same results may be observed when using the ECP.

*New database created but dismounted:

*Issue mount command on new database:

*Database verified as mounted.

In this time the Information Store process was made aware of the newly created database allowing for the mount request from the Replication Service to be processed successfully.

Administrators of Office 365 may have been pleasantly surprised to hear that the large item migration limit for hybrid mailbox moves has increased.  Administrators may now migrate messages up to 150 meg.  Originally when performing a migration any item that was found in the mailbox over 35 meg was either left behind or it was the responsibility of the administrator to preserve and handle this data.  This increase should allow administrations to more easily migrate to Office 365.

Over the last two days I have talked with some customers who observed what appeared to be the limits not applying.  Excited about this new feature these administrators were testing it prior to performing their end user migrations.  For example, when performing a migration of a mailbox with known large items the following was observed in the migration log:

1/19/2015 2:07:44 PM [BY2PR02MB315] A large item was encountered: Item (IPM.Note) Subject:"77 mb", Size: 77.51 MB (81,278,846 bytes), Folder:"Inbox"

Why did the new limits not apply?  Let’s take a look…

When a mailbox exists on premises it is represented in Exchange Online as a mail user object.  A mail user object where the mailbox has never been migrated to Exchange Online does not have a mailbox plan applied.  This can be validated with get-mailUser.

PS C:> Get-MailUser | fl maxSendSize,maxReceiveSize

MaxSendSize    : Unlimited
MaxReceiveSize : Unlimited

When the mailbox is migrated to Exchange online the mail user object is converted to mailbox object.  When this occurs a mailbox plan is applied to the mailbox.  This can be validated with get-mailbox.

PS C:> Get-Mailbox | fl maxSendSize,maxReceiveSize

MaxSendSize    : 35 MB (36,700,160 bytes)
MaxReceiveSize : 36 MB (37,748,736 bytes)

What happens if the mailbox is migrated back to on premises?  When this occurs the mailbox object in Exchange Online is converted back to a mail user object.  In this instance though the mail user object retains the original mailbox policy applied.

PS C:> Get-MailUser| fl maxSendSize,maxReceiveSize

MaxSendSize    : 35 MB (36,700,160 bytes)
MaxReceiveSize : 36 MB (37,748,736 bytes)

How does this affect migrations?  When a mailbox policy has already been applied to an object those settings are enforced for the migration.  If the mailbox that was migrated back to on premises was subsequently used, and large items generated in the mailbox, when the mailbox was migrated back to Exchange Online any large items would be discarded due to the applied mailbox plan limit of 35 meg.  In each of these cases the administrators were testing with mailboxes that has previously been migrated to Exchange Online and then back on premises.

To allow large item limits to migrate changes, were taken to apply the mailbox policy after the mailbox was migrated and control the item size limit as a part of the move mailbox procedure.  For example, this is a mailbox that contained large items > 35 meg and was migrated after the service changes were applied.  The items migrated successfully.  Even after migration the original mailbox policy of 35 meg was applied. 

PS C:> Get-Mailbox | fl maxSendSize,maxReceiveSize

MaxSendSize    : 35 MB (36,700,160 bytes)
MaxReceiveSize : 36 MB (37,748,736 bytes)

This is correct as that is the existing mailbox policy of Exchange Online.

If you are migrating mailboxes and observe large items being left behind you may want to check the mail user object within Exchange Online to see if a mailbox policy has been applied.  Chances are that one has and the object we are dealing with was migrated previously.  If this is the case the mailbox must be treated like before – the large items must either be skipped or retained manually prior to the migration being performed.  You can ensure that no loss occurs by continuing to specify a large item limit of 0 on the migration batches, which will cause the individual move to fail if a large item is encountered.

In recent weeks I have had the chance to work with several customers on an inquiry regarding the SMTP address displayed in an Outlook profile.

Let us take a look at an example.  In Exchange 2013 a user account is provisioned and has a primary SMTP address of alias@domain.com.

When an Outlook profile is configured, the primary address is displayed as the root name of the default folder structure.

Email addresses may be changed at anytime.  In this instance the primary email address of the users account is changed.  It may be changed to an address within the same domain or to another accepted domain.

In this example the primary SMTP address has changed to alias@subdomain.domain.com.

When launching Outlook the user may notice that the email address in the root folder has not changed.

This is by design.  The email address stamped in the root folder is determined at profile creation time and should reflect the primary SMTP address stamped on the user.  Although the address is strictly cosmetic some customers require it be changed if the users primary SMTP address changes.  The only supported method at this time to change this is to delete and recreate the profile for the user. 

In Exchange 2013 we introduced the concept of server component states.  This allows administrators a great deal of flexibility to control a server’s ability to service certain requests either partially or fully.  More information on server component states can be found here.

Normally when applying a cumulative update to an Exchange 2013 instance the server would be put into maintenance mode.  Maintenance mode usually involves two separate commands:

• Executing the StartDagServerMaintenance.ps1 script.
• Setting all the component states on a given server to server wide offline.
• Disabling automatic recovery actions on the server.

This effectively migrates all functionality off of the server ensuring that maintenance can be safely performed.  In Exchange 2013, we have also extended setup to perform some of the maintenance tasks prescribed here – specifically handling server health states to ensure that the server can be safely upgraded.  When performing setup.exe /mode:upgrade to upgrade between Exchange 2013 CUs we now perform the following:

• Set the monitoring state of the server to inactive.
• Prevent automatic recovery actions from occurring on the server.
• Set the ServerWideOffline component state to InActive

This essentially disables all health checking against the server, all automatic recovery actions as a result of that health checking, and prevents the server from performing transport and other client functions.

Let’s examine a sample upgrade.

Prior to running setup.exe /mode:upgrade get-servercomponentstate can be executed.  In almost all cases the components show active:

[PS] C:>Get-ServerComponentState -Identity MBX-TEST

Server                                  Component                               State
——                                  ———                               —–
MBX-TEST.exchange.msft                  ServerWideOffline                       Active
MBX-TEST.exchange.msft                  HubTransport                            Active
MBX-TEST.exchange.msft                  FrontendTransport                       Active
MBX-TEST.exchange.msft                  Monitoring                              Active
MBX-TEST.exchange.msft                  RecoveryActionsEnabled                  Active
MBX-TEST.exchange.msft                  AutoDiscoverProxy                       Active
MBX-TEST.exchange.msft                  ActiveSyncProxy                         Active
MBX-TEST.exchange.msft                  EcpProxy                                Active
MBX-TEST.exchange.msft                  EwsProxy                                Active
MBX-TEST.exchange.msft                  ImapProxy                               Active
MBX-TEST.exchange.msft                  OabProxy                                Active
MBX-TEST.exchange.msft                  OwaProxy                                Active
MBX-TEST.exchange.msft                  PopProxy                                Active
MBX-TEST.exchange.msft                  PushNotificationsProxy                  Active
MBX-TEST.exchange.msft                  RpsProxy                                Active
MBX-TEST.exchange.msft                  RwsProxy                                Active
MBX-TEST.exchange.msft                  RpcProxy                                Active
MBX-TEST.exchange.msft                  UMCallRouter                            Active
MBX-TEST.exchange.msft                  XropProxy                               Active
MBX-TEST.exchange.msft                  HttpProxyAvailabilityGroup              Active
MBX-TEST.exchange.msft                  ForwardSyncDaemon                       Active
MBX-TEST.exchange.msft                  ProvisioningRps                         Active
MBX-TEST.exchange.msft                  MapiProxy                               Active
MBX-TEST.exchange.msft                  EdgeTransport                           Active
MBX-TEST.exchange.msft                  HighAvailability                        Active
MBX-TEST.exchange.msft                  SharedCache                             Active

At this time we execute the setup.exe /mode:upgrade.  A review of the setup log shows that we are building commands to execute to adjust the server component state and disable server health checking.

[10/14/2014 17:30:59.0886] [1] Executing: 
          try
          {
            $Target = $env:COMPUTERNAME
            try
            {
              $exSrv = get-ExchangeServer $Target -ErrorAction SilentlyContinue
            }
            catch
            {
              Write-ExchangeSetupLog -Warning "Unable to set monitoring and server state to inactive.  Setup will continue.";
            }

            if ($exSrv -eq $null)
            {
              Write-ExchangeSetupLog -Warning "$Target is not an Exchange Server. Unable to set monitoring and server state to inactive.  Setup will continue.";
              return
            }

            Set-ServerComponentState $Target -Component Monitoring -Requester Functional -State Inactive
            Write-ExchangeSetupLog -Info "Monitoring has been set to Inactive while setup is running."

            Set-ServerComponentState $Target -Component RecoveryActionsEnabled -Requester Functional -State Inactive
            Write-ExchangeSetupLog -Info "RecoveryActionsEnabled has been set to Inactive while setup is running."

            Set-ServerComponentState $Target -Component ServerWideOffline -Requester Functional -State InActive
            Write-ExchangeSetupLog -Info "The server state has been set to Inactive while setup is running."
          }
          catch
          {
            Write-ExchangeSetupLog -Warning "Unable to set monitoring and server state to inactive.  Setup can not continue.";
            throw;
          }

After the commands are built the setup routine executes them.  It should be noted that these steps are executed directly at the beginning of setup – even before pre-requisite analysis etc.

[10/14/2014 17:31:00.0510] [2] Active Directory session settings for 'Set-ServerComponentState' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-TEST.exchange.msft', Preferred Global Catalog: 'DC-TEST.exchange.msft', Preferred Domain Controllers: '{ DC-TEST.exchange.msft }'
[10/14/2014 17:31:00.0510] [2] User specified parameters:  -Component:'Monitoring' -Requester:'Functional' -State:'Inactive' -Identity:'MBX-TEST'
[10/14/2014 17:31:00.0510] [2] Beginning processing Set-ServerComponentState
[10/14/2014 17:31:02.0339] [2] Ending processing Set-ServerComponentState
[10/14/2014 17:31:02.0354] [2] Beginning processing Write-ExchangeSetupLog
[10/14/2014 17:31:02.0354] [2] Monitoring has been set to Inactive while setup is running.
[10/14/2014 17:31:02.0354] [2] Ending processing Write-ExchangeSetupLog
[10/14/2014 17:31:02.0354] [2] Active Directory session settings for 'Set-ServerComponentState' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-TEST.exchange.msft', Preferred Global Catalog: 'DC-TEST.exchange.msft', Preferred Domain Controllers: '{ DC-TEST.exchange.msft }'
[10/14/2014 17:31:02.0354] [2] User specified parameters:  -Component:'RecoveryActionsEnabled' -Requester:'Functional' -State:'Inactive' -Identity:'MBX-TEST'
[10/14/2014 17:31:02.0354] [2] Beginning processing Set-ServerComponentState
[10/14/2014 17:31:02.0589] [2] Ending processing Set-ServerComponentState
[10/14/2014 17:31:02.0589] [2] Beginning processing Write-ExchangeSetupLog
[10/14/2014 17:31:02.0589] [2] RecoveryActionsEnabled has been set to Inactive while setup is running.
[10/14/2014 17:31:02.0589] [2] Ending processing Write-ExchangeSetupLog
[10/14/2014 17:31:02.0589] [2] Active Directory session settings for 'Set-ServerComponentState' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-TEST.exchange.msft', Preferred Global Catalog: 'DC-TEST.exchange.msft', Preferred Domain Controllers: '{ DC-TEST.exchange.msft }'
[10/14/2014 17:31:02.0589] [2] User specified parameters:  -Component:'ServerWideOffline' -Requester:'Functional' -State:'Inactive' -Identity:'MBX-TEST'
[10/14/2014 17:31:02.0589] [2] Beginning processing Set-ServerComponentState
[10/14/2014 17:31:02.0761] [2] Ending processing Set-ServerComponentState
[10/14/2014 17:31:02.0761] [2] Beginning processing Write-ExchangeSetupLog
[10/14/2014 17:31:02.0761] [2] The server state has been set to Inactive while setup is running.

During setup running get-servercomponentstate reflects that all components are set to inactive.

[PS] C:>Get-ServerComponentState -Identity MBX-TEST

Server                                  Component                               State
——                                  ———                               —–
MBX-TEST.exchange.msft                  ServerWideOffline                       Inactive
MBX-TEST.exchange.msft                  HubTransport                            Inactive
MBX-TEST.exchange.msft                  FrontendTransport                       Inactive
MBX-TEST.exchange.msft                  Monitoring                              Inactive
MBX-TEST.exchange.msft                  RecoveryActionsEnabled                  Inactive
MBX-TEST.exchange.msft                  AutoDiscoverProxy                       Inactive
MBX-TEST.exchange.msft                  ActiveSyncProxy                         Inactive
MBX-TEST.exchange.msft                  EcpProxy                                Inactive
MBX-TEST.exchange.msft                  EwsProxy                                Inactive
MBX-TEST.exchange.msft                  ImapProxy                               Inactive
MBX-TEST.exchange.msft                  OabProxy                                Inactive
MBX-TEST.exchange.msft                  OwaProxy                                Inactive
MBX-TEST.exchange.msft                  PopProxy                                Inactive
MBX-TEST.exchange.msft                  PushNotificationsProxy                  Inactive
MBX-TEST.exchange.msft                  RpsProxy                                Inactive
MBX-TEST.exchange.msft                  RwsProxy                                Inactive
MBX-TEST.exchange.msft                  RpcProxy                                Inactive
MBX-TEST.exchange.msft                  UMCallRouter                            Inactive
MBX-TEST.exchange.msft                  XropProxy                               Inactive
MBX-TEST.exchange.msft                  HttpProxyAvailabilityGroup              Inactive
MBX-TEST.exchange.msft                  ForwardSyncDaemon                       Inactive
MBX-TEST.exchange.msft                  ProvisioningRps                         Inactive
MBX-TEST.exchange.msft                  MapiProxy                               Inactive
MBX-TEST.exchange.msft                  EdgeTransport                           Inactive
MBX-TEST.exchange.msft                  HighAvailability                        Inactive
MBX-TEST.exchange.msft                  SharedCache                             Inactive

In almost all cases this is very helpful as it helps to ensure that the server is not actively servicing clients or subject to healthy monitoring while services may be in a disabled or upgrading state.  Where this can cause issues though – what happens if setup fails?  For example, recently a customer presented a case where the pre-requisite tests were failing.  The fix for the pre-requisite could not be immediately acted on so the customer deferred the action to another maintenance window.  They then discovered that the components remained in an inactive state meaning the server was not fully functional.  To the customer this was unexpected – to support this was fully expected.  As previously indicated the component states are adjusted very early in setup, therefore if any actions during setup fail the components remain inactive.  When setups completes successfully the component states are reverted.

In this example I have executed setup.exe /mode:upgrade from the command line.  During the pre-requisite analysis I performed a CTRL-C to cause setup to abort. 

PS C:CU6> .setup.exe /mode:upgrade /iacceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 6 Unattended Setup
Copying Files…
File copy complete. Setup will now collect additional information needed for installation.
Languages
Mailbox role: Transport service
Client Access role: Front End Transport service
Mailbox role: Client Access service
Mailbox role: Unified Messaging service
Mailbox role: Mailbox service
Management tools
Client Access role: Client Access Front End service

Performing Microsoft Exchange Server Prerequisite Check

    Configuring Prerequisites                                                                         COMPLETED
    Prerequisite Analysis                                                                             84%

When this was completed I gathered the component states and noted they continue to remain inactive.

[PS] C:>Get-ServerComponentState -Identity MBX-TEST

Server                                  Component                               State
——                                  ———                               —–
MBX-TEST.exchange.msft                  ServerWideOffline                       Inactive
MBX-TEST.exchange.msft                  HubTransport                            Inactive
MBX-TEST.exchange.msft                  FrontendTransport                       Inactive
MBX-TEST.exchange.msft                  Monitoring                              Inactive
MBX-TEST.exchange.msft                  RecoveryActionsEnabled                  Inactive
MBX-TEST.exchange.msft                  AutoDiscoverProxy                       Inactive
MBX-TEST.exchange.msft                  ActiveSyncProxy                         Inactive
MBX-TEST.exchange.msft                  EcpProxy                                Inactive
MBX-TEST.exchange.msft                  EwsProxy                                Inactive
MBX-TEST.exchange.msft                  ImapProxy                               Inactive
MBX-TEST.exchange.msft                  OabProxy                                Inactive
MBX-TEST.exchange.msft                  OwaProxy                                Inactive
MBX-TEST.exchange.msft                  PopProxy                                Inactive
MBX-TEST.exchange.msft                  PushNotificationsProxy                  Inactive
MBX-TEST.exchange.msft                  RpsProxy                                Inactive
MBX-TEST.exchange.msft                  RwsProxy                                Inactive
MBX-TEST.exchange.msft                  RpcProxy                                Inactive
MBX-TEST.exchange.msft                  UMCallRouter                            Inactive
MBX-TEST.exchange.msft                  XropProxy                               Inactive
MBX-TEST.exchange.msft                  HttpProxyAvailabilityGroup              Inactive
MBX-TEST.exchange.msft                  ForwardSyncDaemon                       Inactive
MBX-TEST.exchange.msft                  ProvisioningRps                         Inactive
MBX-TEST.exchange.msft                  MapiProxy                               Inactive
MBX-TEST.exchange.msft                  EdgeTransport                           Inactive
MBX-TEST.exchange.msft                  HighAvailability                        Inactive
MBX-TEST.exchange.msft                  SharedCache                             Inactive

In this case if no action was taken these states would be maintained until:

• The administrator completed the setup operation.
• The administrator reverted the component states.

In this case the administrator reverted the component states by performing the following actions:

• Set-ServerComponentState <SERVER> –component Monitoring –requester Functional –state Active
• Set-ServerComponentState <SERVER> –component RecoveryActionsEnabled –requestor Functional –state Active
• Set-ServerComponentState <SERVER> –component ServerWideOffline –requestor Functional –state Active

This combined with StopDagServerMaintenance.ps1 (if utilized) would revert the maintenance actions performed by both the administrator and automatically by setup.

Had the administrator allowed setup to complete successfully these actions would have been reverted automatically upon successful completion of setup.  In this instance the setup was successful:

PS C:CU6> .setup.exe /mode:upgrade /iacceptExchangeServerLicenseTerms

Welcome to Microsoft Exchange Server 2013 Cumulative Update 6 Unattended Setup
Copying Files…
File copy complete. Setup will now collect additional information needed for installation.
Languages
Mailbox role: Transport service
Client Access role: Front End Transport service
Mailbox role: Client Access service
Mailbox role: Unified Messaging service
Mailbox role: Mailbox service
Management tools
Client Access role: Client Access Front End service

Performing Microsoft Exchange Server Prerequisite Check

    Configuring Prerequisites                                                                         COMPLETED
    Prerequisite Analysis                                                                             COMPLETED
Setup can't detect a Send connector with an address space of '*'. Mail flow to the Internet may not work properly.
For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.NoConnectorToStar.aspx

Configuring Microsoft Exchange Server

    Organization Preparation                                                                          COMPLETED
    Preparing Setup                                                                                   COMPLETED
    Stopping Services                                                                                 COMPLETED
    Language Files                                                                                    COMPLETED
    Removing Exchange Files                                                                           COMPLETED
    Preparing Files                                                                                   COMPLETED
    Copying Exchange Files                                                                            COMPLETED
    Language Files                                                                                    COMPLETED
    Restoring Services                                                                                COMPLETED
    Language Configuration                                                                            COMPLETED
    Mailbox role: Transport service                                                                   COMPLETED
    Client Access role: Front End Transport service                                                   COMPLETED
    Mailbox role: Client Access service                                                               COMPLETED
    Mailbox role: Unified Messaging service                                                           COMPLETED
    Mailbox role: Mailbox service                                                                     COMPLETED
    Exchange Management Tools                                                                         COMPLETED
    Client Access role: Client Access Front End service                                               COMPLETED
    Finalizing Setup                                                                                  COMPLETED

The Exchange Server setup operation completed successfully.

When reviewing the server component states it is observed that everything is active:

[PS] C:>Get-ServerComponentState -Identity MBX-TEST

Server                                  Component                               State
——                                  ———                               —–
MBX-TEST.exchange.msft                  ServerWideOffline                       Active
MBX-TEST.exchange.msft                  HubTransport                            Active
MBX-TEST.exchange.msft                  FrontendTransport                       Active
MBX-TEST.exchange.msft                  Monitoring                              Active
MBX-TEST.exchange.msft                  RecoveryActionsEnabled                  Active
MBX-TEST.exchange.msft                  AutoDiscoverProxy                       Active
MBX-TEST.exchange.msft                  ActiveSyncProxy                         Active
MBX-TEST.exchange.msft                  EcpProxy                                Active
MBX-TEST.exchange.msft                  EwsProxy                                Active
MBX-TEST.exchange.msft                  ImapProxy                               Active
MBX-TEST.exchange.msft                  OabProxy                                Active
MBX-TEST.exchange.msft                  OwaProxy                                Active
MBX-TEST.exchange.msft                  PopProxy                                Active
MBX-TEST.exchange.msft                  PushNotificationsProxy                  Active
MBX-TEST.exchange.msft                  RpsProxy                                Active
MBX-TEST.exchange.msft                  RwsProxy                                Active
MBX-TEST.exchange.msft                  RpcProxy                                Active
MBX-TEST.exchange.msft                  UMCallRouter                            Active
MBX-TEST.exchange.msft                  XropProxy                               Active
MBX-TEST.exchange.msft                  HttpProxyAvailabilityGroup              Active
MBX-TEST.exchange.msft                  ForwardSyncDaemon                       Active
MBX-TEST.exchange.msft                  ProvisioningRps                         Active
MBX-TEST.exchange.msft                  MapiProxy                               Active
MBX-TEST.exchange.msft                  EdgeTransport                           Active
MBX-TEST.exchange.msft                  HighAvailability                        Active
MBX-TEST.exchange.msft                  SharedCache                             Active

In the setup log it is observed that the setup routine constructs the correct commands to reverse the inactive setting.

[10/14/2014 19:45:47.0400] [1] Executing:
            $Target = $env:COMPUTERNAME
            $expectSrv = $true
            $Error.Clear()
            try
            {
              $exSrv = get-ExchangeServer $Target -ErrorAction Stop
            }
            catch
            {
              if ($Error.Exception.GetType().Name -eq 'ManagementObjectNotFoundException')
              {
                $expectSrv = $false;
                Write-ExchangeSetupLog -Warning "$Target is not an Exchange Server.  Unable to set monitoring and server state to active.  Setup will continue.";
              }
              else
              {
                Write-ExchangeSetupLog -Error "SetServerStateForSetup fails for $Target due to $Error";
              }
            }

            if ($exSrv -eq $null)
            {
              if (!$expectSrv)
              {
                return
              }
              throw "SetServerStateForSetup fails due to $Error";
            }

            Set-ServerComponentState $Target -Component Monitoring -Requester Functional -State Active
            Write-ExchangeSetupLog -Info "Install is complete.  Monitoring has been set to Active.";

            Set-ServerComponentState $Target -Component RecoveryActionsEnabled -Requester Functional -State Active
            Write-ExchangeSetupLog -Info "Install is complete.  RecoveryActionsEnabled has been set to Active.";

            Set-ServerComponentState $Target -Component ServerWideOffline -Requester Functional -State Active
            Write-ExchangeSetupLog -Info "Install is complete.  Server state has been set to Active.";

These commands are then executed at the end of setup during finalization.

[10/14/2014 19:45:47.0854] [2] Active Directory session settings for 'Set-ServerComponentState' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-TEST.exchange.msft', Preferred Global Catalog: 'DC-TEST.exchange.msft', Preferred Domain Controllers: '{ DC-TEST.exchange.msft }'
[10/14/2014 19:45:47.0854] [2] User specified parameters:  -Component:'Monitoring' -Requester:'Functional' -State:'Active' -Identity:'MBX-TEST'
[10/14/2014 19:45:47.0854] [2] Beginning processing Set-ServerComponentState
[10/14/2014 19:45:48.0166] [2] Ending processing Set-ServerComponentState
[10/14/2014 19:45:48.0182] [2] Beginning processing Write-ExchangeSetupLog
[10/14/2014 19:45:48.0182] [2] Install is complete.  Monitoring has been set to Active.
[10/14/2014 19:45:48.0182] [2] Ending processing Write-ExchangeSetupLog
[10/14/2014 19:45:48.0182] [2] Active Directory session settings for 'Set-ServerComponentState' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-TEST.exchange.msft', Preferred Global Catalog: 'DC-TEST.exchange.msft', Preferred Domain Controllers: '{ DC-TEST.exchange.msft }'
[10/14/2014 19:45:48.0182] [2] User specified parameters:  -Component:'RecoveryActionsEnabled' -Requester:'Functional' -State:'Active' -Identity:'MBX-TEST'
[10/14/2014 19:45:48.0182] [2] Beginning processing Set-ServerComponentState
[10/14/2014 19:45:48.0620] [2] Ending processing Set-ServerComponentState
[10/14/2014 19:45:48.0620] [2] Beginning processing Write-ExchangeSetupLog
[10/14/2014 19:45:48.0635] [2] Install is complete.  RecoveryActionsEnabled has been set to Active.
[10/14/2014 19:45:48.0635] [2] Ending processing Write-ExchangeSetupLog
[10/14/2014 19:45:48.0635] [2] Active Directory session settings for 'Set-ServerComponentState' are: View Entire Forest: 'True', Configuration Domain Controller: 'DC-TEST.exchange.msft', Preferred Global Catalog: 'DC-TEST.exchange.msft', Preferred Domain Controllers: '{ DC-TEST.exchange.msft }'
[10/14/2014 19:45:48.0635] [2] User specified parameters:  -Component:'ServerWideOffline' -Requester:'Functional' -State:'Active' -Identity:'MBX-TEST'
[10/14/2014 19:45:48.0635] [2] Beginning processing Set-ServerComponentState
[10/14/2014 19:45:48.0870] [2] Ending processing Set-ServerComponentState
[10/14/2014 19:45:48.0870] [2] Beginning processing Write-ExchangeSetupLog
[10/14/2014 19:45:48.0870] [2] Install is complete.  Server state has been set to Active.
[10/14/2014 19:45:48.0870] [2] Ending processing Write-ExchangeSetupLog

When the setup routine completes successfully the inactive states are correctly reset.  Should an error occur anytime after setup is initialized, and setup has not completed, it may be necessary for the administrator to manually revert these settings.

At any given time, in every database availability group (DAG), there is one member that is responsible for the coordination of database actions across the DAG. This member is known as the Primary Active Manager (PAM). The current PAM can be determined by using Get-DatabaseAvailabilityGroup –Status, as shown below.

[PS] C:>Get-DatabaseAvailabilityGroup -Status -Identity DAG | fl name,primaryActiveManager

Name                 : DAG
PrimaryActiveManager : MBX-1

The mailbox server that is the PAM is always the current owner of the Cluster Core Resource group.

[PS] C:>Get-ClusterGroup -Cluster MBX-1

Name                                    OwnerNode                               State
—-                                    ———                               —–
Available Storage                       MBX-3                                   Offline
Cluster Group                           MBX-1                                   Online

The cluster group may contain several cluster resources. The PAM does not depend on the state of any of the resources in this group, and the PAM role will always be assigned to the node that owns the Cluster Group.

The Cluster Group can be moved between members using the cluster management tools. 

Windows 2008 R2 / Windows 2012 / Windows 2012 R2:

Command Line:

Cluster.exe cluster <DAGNAME> group “Cluster Group” /moveto:<NODE>

Powershell:

Move-ClusterGroup –cluster <DAGNAME> –name “Cluster Group” –node:<NODE>

*Note:  In Windows Server 2012 and Windows Server 2012 R2, Powershell is the preferred way to manage clusters.

Each DAG member that does not own the cluster group is a Standby Active Manager (SAM).  When the cluster group is moved between nodes, a notification process detects that the Cluster Group owner has changed. This triggers detection logic to determine the new PAM.  In this example, the cluster group is moved from MBX-1 to MBX-2.

PS C:> Move-ClusterGroup -Name "Cluster Group" -Node MBX-2

Name                                    OwnerNode                               State
—-                                    ———                               —–
Cluster Group                           MBX-2                                   Online

A review of Microsoft-Exchange-HighAvailability/Operational crimson channel shows the promotion of MBX-2 from SAM to PAM:

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      227
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-2.domain.com
Description:
Active manager configuration change detected. (PreviousRole='SAM', CurrentRole='PAM', ChangeFlags='Role, CurrentPAM', LastError='<none>')

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:48 AM
Event ID:      111
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-2.domain.com
Description:
Active manager role changed from 'SAM' to 'PAM'

A review of the Microsoft-Exchange-HighAvailability/Operational crimson channel shows the demotion of MBX-1 from PAM to SAM:

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      227
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-1.domain.com
Description:
Active manager configuration change detected. (PreviousRole='PAM', CurrentRole='SAM', ChangeFlags='Role, CurrentPAM', LastError='<none>')

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      111
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-1.domain.com
Description:
Active manager role changed from 'PAM' to 'SAM'

Other servers within the DAG also acknowledge the move but continue to maintain their existing roles:

Log Name:      Microsoft-Exchange-HighAvailability/Operational
Source:        Microsoft-Exchange-HighAvailability
Date:          8/3/2014 8:37:47 AM
Event ID:      227
Task Category: Role Monitoring
Level:         Information
Keywords:     
User:          SYSTEM
Computer:      MBX-3.domain.com
Description:
Active manager configuration change detected. (PreviousRole='SAM', CurrentRole='SAM', ChangeFlags='CurrentPAM', LastError='<none>')

The Cluster service is also responsible for automatic arbitration of this group.  Automatic arbitration may occur for a number of reasons including: 

• The failure of a member
• The failure of a resource contained within the Cluster group

In most cases, Exchange administrators should not be concerned with the owner of the cluster group or the node designated as the PAM.  This is true even for DAGs that span multiple sites where the PAM may be a node in a distant datacenter.

In recent weeks I have fielded several questions from administrators that are concerned with which member owns the PAM. Questions like:

• Should I set a preferred owner on the cluster group so that nodes in my primary datacenter are preferred over my disaster recovery datacenter?
• How do I prevent a server in the disaster recovery datacenter from becoming the PAM?
• Can I pause my nodes in cluster to prevent them from becoming the PAM?
• Should I remove possible owner on the cluster name to prevent it from coming online on a server in my disaster recovery datacenter?

All of these questions require modifying properties of the cluster core resource group.  By default Exchange establishes the desired settings on the cluster core resource group.  Modifying these settings is typically not necessary and can sometimes cause undesired results.  For example, recently a customer paused all of the members in a disaster recovery datacenter to prevent those servers from becoming the PAM.  This worked very well in preventing arbitration of the cluster core resource group to these nodes, until instability of the Cluster service in the primary datacenter resulted in no members being able to take ownership the cluster core resources.  In this instance the PAM was lost, and coordination of database activities across the DAG failed. 

The Cluster service is designed to allow the cluster group to arbitrate freely across nodes.  Attempts to modify the failover behavior, or prevent failure between nodes, can yield undesired results and potential instability.  As a result, we recommend that you leave the default out-of-box settings intact, unless you are directed by Microsoft Customer Services and Support to change them.

Although Exchange no longer uses shared storage, it can be deployed in clustered environments where shared storage is used. The storage is not actually shared between nodes as was the case with traditional clustering. Instead, the storage is presented through traditional shared storage controllers including fibre channel and iSCSI.

When creating a cluster with Windows Server 2012 or Windows Server 2012 R2, storage found on a shared bus is not automatically added to the cluster. When adding a node to an existing cluster the administrator is presented with an option to add shared storage automatically.  

PS C:> Add-ClusterNode -Cluster TEST -Name MBX-2

Allowing this option to remain checked, which is the default, will result in the Cluster service automatically adding all storage found on a shared storage bus to the cluster and clustered storage (even for disks that are not shared between nodes).  The same behavior is observed when using the Add-ClusterNode cmdlet. 

The disks can be observed within Failover Cluster Manager:

Physical disk resources can also be reviewed using Get-ClusterResource:

PS C:UsersAdministrator.EXCHANGE> Get-ClusterResource

Name                          State                         OwnerGroup                    ResourceType
—-                          —–                         ———-                    ————
Cluster Disk 1                Online                        Cluster Group                 Physical Disk
Cluster Disk 2                Online                        Available Storage             Physical Disk
Cluster Disk 3                Offline                       Available Storage             Physical Disk
Cluster Disk 4                Offline                       Available Storage             Physical Disk
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

Administrators can prevent the addition of the shared disks by unchecking “Add all eligible storage to the cluster” or by using the –NoStorage option with Add-ClusterNode.

PS C:> Add-ClusterNode -Cluster TEST -Name MBX-2 -NoStorage
Report file location: C:WindowsclusterReportsAdd Node Wizard 6252c9cd-5117-474b-bb7f-d117a98759ee on 2014.07.20 At 05.31.26.mht

This configuration can be validated using Get-ClusterResource.

PS C:UsersAdministrator.EXCHANGE> Get-ClusterResource

Name                          State                         OwnerGroup                    ResourceType
—-                          —–                         ———-                    ————
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

With Windows Server 2008 R2, storage found on a shared bus is not automatically added to a cluster during creation or when a node is added. The behavior is the same whether you are creating the cluster or adding the node with Failover Cluster Manager or with PowerShell. The confirmation dialog has no “Add all eligible storage to the cluster” option.

Additionally the Add-ClusterNode cmdlet does not have a –NoStorage option. 

PS C:> Add-ClusterNode -Cluster Cluster -Name Node-2 -noStorage
Add-ClusterNode : A parameter cannot be found that matches parameter name 'noStorage'.
At line:1 char:57
+ Add-ClusterNode -Cluster Cluster -Name Node-2 -noStorage <<<<
    + CategoryInfo          : InvalidArgument: (:) [Add-ClusterNode], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Microsoft.FailoverClusters.PowerShell.AddClusterNodeCommand

As the membership is modified within the cluster, the lack of clustered disks can be validated with Get-ClusterResource, as well as with Failover Cluster Manager.

PS C:> Get-ClusterResource -Cluster Cluster

Name                          State                         Group                         ResourceType
—-                          —–                         —–                         ————
Cluster IP Address            Online                        Cluster Group                 IP Address
Cluster Name                  Online                        Cluster Group                 Network Name

At this point, you’re probably wondering why I am writing a blog post about shared storage and Exchange, since database availability groups (DAGs) don’t use shared storage.

Over the course of the last few weeks, I have reviewed some DAG configurations where physical disk resources exist within the cluster. This is not a desired configuration. When disks are added to the cluster, it is the responsibility of the cluster disk driver to manage access to these resources. In these cases, checking Disk Management shows that the disks have a status of reserved. This status indicates that the storage is no longer under the control of Windows, but is instead being managed by the cluster disk driver.

Overall this causes several issues. For example, if the Cluster service fails for any reason, this makes storage in accessible to Exchange. The drive letters and mount point mappings are the same across each node even though they do not match the same physical disk. This causes confusion within the cluster and it can lead to storage instability.

Correcting this condition is as simple as removing the physical disk resources from cluster. This can be done using either Failover Cluster Manager or PowerShell. I recommend performing this operation during a maintenance period as it can result in the storage being temporarily inaccessible while it transitions from cluster control to Windows partition manager control.

At this point, you’re probably wondering how a DAG’s cluster can end up with storage shared that is controlled by the cluster.

There are actually a couple of causes. In some cases, it happens because the Exchange cmdlets failed (for example, Add-DatabaseAvailabilityGroupServer fails to successfully add a DAG member). In other cases, it happens because the cluster is rebuilt as part of a site activation process. When using Failover Cluster Manager to perform this operation you must ensure that the “Add all eligible storage to the cluster” option is unchecked in Failover Cluster Manager or that you use the –NoStorage option in PowerShell.

In the cases I have been involved with, it was determined that cluster membership was adjusted using Failover Cluster Manager without unchecking the add storage option. It is important for administrators to be aware of this new default option and ensure that if this condition is encountered, it is corrected as soon as possible.