In Exchange 2013 we have introduced managed availability.  The managed availability process runs within the Microsoft Exchange Health Manager service (MSExchangeHMHost.exe).

 

After booting a server it has been observed that the Microsoft Exchange Health Manager service fails to start automatically.  Attempts to start the service manually are successful.  When reviewing the system log the following events are present:

 

Log Name:      System
Source:        Service Control Manager
Date:          11/13/2013 9:23:47 AM
Event ID:      7009
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-1.domain.com
Description:
A timeout was reached (30000 milliseconds) while waiting for the Microsoft Exchange Health Manager service to connect.

Log Name:      System
Source:        Service Control Manager
Date:          11/13/2013 9:23:47 AM
Event ID:      7000
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      MBX-1.domain.com
Description:
The Microsoft Exchange Health Manager service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.

 

To correct the issue the startup type for the service can be changed from AUTOMATIC to AUTOMATIC (DELAYED). 

 

 

After setting the startup type of the service to AUTOMATIC (DELAYED) the following event is noted in the system log on boot:

 

Log Name:      System
Source:        Service Control Manager
Date:          11/13/2013 10:13:26 AM
Event ID:      7036
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      MBX-1.domain.com
Description:
The Microsoft Exchange Health Manager service entered the running state.

 

This event indicates the Microsoft Exchange Health Manager service started successfully.

As you may know, Office 365 supports a number of different client protocols, including POP and IMAP and a variety of POP and IMAP clients. By default, POP or IMAP clients will be configured to use Outlook Web App (OWA) for handling calendar invitations. When these clients receive a meeting request, within the body of the invite is a link. Clicking on the link allows the user to open their mailbox via OWA so they can accept or decline the request.

 

If you are using a POP or IMAP client that is capable of handling ICAL messages, you may want to change your configuration settings so that you can get a better experience.

 

You can configure – on a per-mailbox basis – how POP and IMAP clients receive calendar appointments.  Get-CASMailbox can be used to view your current settings.

 

Get-CASMAILBOX –identity <NAME> | fl name,*pop*,*imap*

Name                                    : administrator
ExternalPopSettings                     :
InternalPopSettings                     :
PopEnabled                              : True
PopUseProtocolDefaults                  : True
PopMessagesRetrievalMimeFormat          : BestBodyFormat
PopEnableExactRFC822Size                : False
PopSuppressReadReceipt                  : False
PopForceICalForCalendarRetrievalOption  : False
ExternalImapSettings                    :
InternalImapSettings                    :
ImapEnabled                             : True
ImapUseProtocolDefaults                 : True
ImapMessagesRetrievalMimeFormat         : BestBodyFormat
ImapEnableExactRFC822Size               : False
ImapSuppressReadReceipt                 : False
ImapForceICalForCalendarRetrievalOption : False

In the above output, you can see two attributes that you need to set to True to enable iCAL support. The *ICALForCalendarRetrievalOption specifies that the client should be provided calendar appointments that are in ICAL format.

 

Also note the *UseProtocolDefaults attributes.  These must be set to False in order for any changes to *ICALForCalendarRetrievalOption to take effect.  You can use Set-CASMailbox to change these settings: 

 

Set-CASMailbox –identity <NAME> –PopUseProtocolDefaults:$FALSE –ImapUseProtocolDefaults:$FALSE –PopForceICalForCalendarRetrievalOption:$TRUE –ImapForceICalForCalendarRetrievalOption:$TRUE

 

Name                                    : administrator
ExternalPopSettings                     :
InternalPopSettings                     :
PopEnabled                              : True
PopUseProtocolDefaults                  : False
PopMessagesRetrievalMimeFormat          : BestBodyFormat
PopEnableExactRFC822Size                : False
PopSuppressReadReceipt                  : False
PopForceICalForCalendarRetrievalOption  : True
ExternalImapSettings                    :
InternalImapSettings                    :
ImapEnabled                             : True
ImapUseProtocolDefaults                 : False
ImapMessagesRetrievalMimeFormat         : BestBodyFormat
ImapEnableExactRFC822Size               : False
ImapSuppressReadReceipt                 : False
ImapForceICalForCalendarRetrievalOption : True

The settings described here are per-mailbox settings.  There is no global setting to change the default for the entire tenant. There is also no method to adjust the settings per-client.

In Exchange Server 2013 Cumulative Update 2 (CU2) we introduced support for a maximum of 100 mounted mailbox databases per server (active or passive) when an Enterprise Edition license is applied.  This was an increase from Exchange Server 2013 RTM and Exchange Server 2013 Cumulative Update 1 (CU1).

 

It has been recently discovered that when updating to CU2 from Exchange Server 2013 RTM or Exchange Server 2013 CU1, the database limit is not increased to 100. This results in the inability to mount more than 50 databases on a server running the Enterprise Edition of Exchange Server 2013. When installing Exchange Server 2013 CU2 directly, the database limit is applied correctly.

 

The Core Problem

 

In this example, I deployed a new install of Exchange Server 2013 CU1 on Windows Server 2012 (although, the operating system doesn’t matter here). After the Setup completed, I performing an LDP dump of the information store object, and noted a database limit of 5.

 

Expanding base ‘CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft’…
Getting 1 entries:
Dn: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = (  );
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 5;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 5;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 52c2fc98-b8b4-4d33-b35b-ca00cb3fcfff;
showInAdvancedViewOnly: TRUE;
uSNChanged: 24300;
uSNCreated: 24300;
whenChanged: 10/27/2013 3:58:35 PM Pacific Daylight Time;
whenCreated: 10/27/2013 3:58:35 PM Pacific Daylight Time;

———–

 

This is normal and expected because all unlicensed installations (e.g., Trial) are treated as Standard Edition, and Standard Edition can have a maximum of 5 mounted databases.

 

Next, I entered an Enterprise Edition license key. This updated the value msExchMaxStoresTotal to a value of 50, which is the expected value for the Enterprise Edition of Exchange Server 2013 RTM and Exchange Server 2013 CU1.

 

Expanding base ‘CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft’…
Getting 1 entries:
Dn: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = (  );
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 100;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 50;
msExchMinAdminVersion: -2147453113;
msExchVersion: 4535486012416;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 52c2fc98-b8b4-4d33-b35b-ca00cb3fcfff;
showInAdvancedViewOnly: TRUE;
uSNChanged: 27284;
uSNCreated: 24300;
whenChanged: 10/28/2013 4:44:30 AM Pacific Daylight Time;
whenCreated: 10/27/2013 3:58:35 PM Pacific Daylight Time;

———–

 

Next, I upgraded the server to Exchange Server 2013 CU2. Once Setup completed, I reviewed the value of msExchMaxStoresTotal, and found that it did not get updated as expected.

 

Expanding base ‘CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft’…
Getting 1 entries:
Dn: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = (  );
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 100;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 50;
msExchMinAdminVersion: -2147453113;
msExchVersion: 4535486012416;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 52c2fc98-b8b4-4d33-b35b-ca00cb3fcfff;
showInAdvancedViewOnly: TRUE;
uSNChanged: 27284;
uSNCreated: 24300;
whenChanged: 10/28/2013 4:44:30 AM Pacific Daylight Time;
whenCreated: 10/27/2013 3:58:35 PM Pacific Daylight Time;

———–

 

In this state, when the administrator tries to mount more than 50 databases (any combination of active or passive) on this server, they will receive an error message:

 

[PS] C:>Mount-Database DB51
Couldn’t mount the database that you specified. Specified database: DB51; Error code: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)

Diagnostic context:
    Lid: 65256
    Lid: 10722   StoreEc: 0x8004060E
    Lid: 1494    —- Remote Context Beg —-
    Lid: 37952   dwParam: 0x144489A
    Lid: 39576   StoreEc: 0x977
    Lid: 35200   dwParam: 0x39F8
    Lid: 58864   StoreEc: 0x8004060E
    Lid: 43248   StoreEc: 0x8004060E
    Lid: 48432   StoreEc: 0x8004060E
    Lid: 54336   dwParam: 0x14448C8
    Lid: 1750    —- Remote Context End —-
    Lid: 1047    StoreEc: 0x8004060E [Database: DB51, Server: TEST-MBX-0.exchange.msft].
    + CategoryInfo          : InvalidOperation: (DB51:ADObjectId) [Mount-Database], InvalidOperationException
    + FullyQualifiedErrorId : [Server=TEST-MBX-0,RequestId=797928f4-fe83-4e1f-8545-e01a72aaf79d,TimeStamp=10/28/2013 7
   :53:32 PM] 5FE3BF1B,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase
    + PSComputerName        : test-mbx-0.exchange.msft

 

In addition, three events will be logged to the Application event log:

 

Log Name:      Application
Source:        MSExchangeIS
Date:          10/28/2013 1:00:27 PM
Event ID:      40003
Task Category: High Availability
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Test-MBX-0.exchange.msft
Description:
Exceeded the max number of 50 databases on this server.

Log Name:      Application
Source:        MSExchangeRepl
Date:          10/28/2013 1:00:27 PM
Event ID:      3154
Task Category: Service
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Test-MBX-0.exchange.msft
Description:
Active Manager failed to mount database DB51 on server Test-MBX-0.exchange.msft. Error: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)
Diagnostic context:
    Lid: 65256 
    Lid: 10722   StoreEc: 0x8004060E
    Lid: 1494    —- Remote Context Beg —-
    Lid: 37952   dwParam: 0x14A9D25
    Lid: 39576   StoreEc: 0x977    
    Lid: 58864   StoreEc: 0x8004060E
    Lid: 43248   StoreEc: 0x8004060E
    Lid: 48432   StoreEc: 0x8004060E
    Lid: 54336   dwParam: 0x14A9D54
    Lid: 1750    —- Remote Context End —-
    Lid: 1047    StoreEc: 0x8004060E

Log Name:      Application
Source:        ExchangeStoreDB
Date:          10/28/2013 1:00:27 PM
Event ID:      226
Task Category: Database recovery
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Test-MBX-0.exchange.msft
Description:
At ’10/28/2013 1:00:27 PM’ the copy of database ‘DB51’ on this server didn’t mount because the number of mailbox database copies on this server exceeds the supported limit. The error returned by failover was "There is only one copy of this mailbox database (DB51). Automatic recovery is not available.".

 

Similar errors and events will occur when attempting to activate a passive database copy on a server with 50 mounted databases or when adding a passive database copy to a server that already has 50 database copies assigned.

 

How to Correct this Condition

 

Fortunately, fixing this problem is pretty easy. You simply re-enter your Enterprise Edition product key on the server, and restart the Microsoft Exchange Information Store service.

 

Moving Forward

 

For the immediate future, we expect this problem to exist when upgrading from Exchange Server 2013 CU1 to any future CU. However, once the server is re-licensed on CU2 or newer, the condition will remain resolved.