Problem
You’ve received complaints that users searching their inbox with OWA or Outlook in Online mode only returns 1 month of results. Using the following Get-MailboxDatabaseCopyStatus cmdlet:
Get-MailboxDatabaseCopyStatus –server <mailboxServerName> | FL *index*,*ma*ser*,databasename
… shows that the ContentIndexState is listed as Healthy for the mailbox databases.
You proceed to stop the following services:
- Microsoft Exchange Search Host Controller
- Microsoft Exchange Search
Then rename or delete the content index folder named with the GUID of the database and restart the services again forcing Exchange to rebuild the content indexes. However, you notice that searching still continues to return the same incomplete results.
Solution
This issue took me a bit of time to troubleshoot because attempting to search for anything related to searching points to the solution above but the environment I was working on did not have the ContentIndexState listed as FailedAndSuspended. I tried searching for the ContentIndexRetryQueueSize variable because the value was high and all of the results pointed me to install CU7 when I already had CU8 installed.
What I found that eventually led me to the underlying issue was the following warning that got repeatedly written to the application log:
Log Name: Application
Source: MSExchangeFastSearch
Event ID: 1009
Level: Warning
The indexing of mailbox database Admin encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. —> Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: 8f76b2d9-77dd-44e6-a8ef-73d2a2539ae1. —> Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)
Diagnostic context:
Lid: 49384
Lid: 51176 StoreEc: 0x476
Lid: 40680 StoreEc: 0x476
Lid: 43980
Lid: 16354 StoreEc: 0x476
Lid: 38985 StoreEc: 0x476
Lid: 20098
Lid: 20585 StoreEc: 0x476
at Microsoft.Mapi.MapiExceptionHelper.InternalThrowIfErrorOrWarning(String message, Int32 hresult, Boolean allowWarnings, Int32 ec, DiagnosticContext diagCtx, Exception innerException)
at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, IExInterface iUnknown, Exception innerException)
at Microsoft.Mapi.MapiEventManager.ReadEvents(Int64 startCounter, Int32 eventCountWanted, Int32 eventCountToCheck, Restriction filter, ReadEventsFlags flags, Boolean includeSid, Int64& endCounter)
at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.<>c__DisplayClass3.<ReadEvents>b__1()
at Microsoft.Exchange.Search.Mdb.MapiUtil.<>c__DisplayClass1`1.<TranslateMapiExceptionsWithReturnValue>b__0()
at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
— End of inner exception stack trace —
at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptions(IDiagnosticsSession tracer, LocalizedString errorString, Action mapiCall)
at Microsoft.Exchange.Search.Mdb.MapiUtil.TranslateMapiExceptionsWithReturnValue[TReturnValue](IDiagnosticsSession tracer, LocalizedString errorString, Func`1 mapiCall)
at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadEvents(Int64 startCounter, Int32 eventCountWanted, ReadEventsFlags flags, Int64& endCounter)
at Microsoft.Exchange.Search.Mdb.NotificationsEventSource.ReadFirstEventCounter()
at Microsoft.Exchange.Search.Engine.NotificationsEventSourceInfo..ctor(IWatermarkStorage watermarkStorage, INotificationsEventSource eventSource, IDiagnosticsSession diagnosticsSession, MdbInfo mdbInfo)
at Microsoft.Exchange.Search.Engine.SearchFeedingController.DetermineFeederStateAndStartFeeders()
at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart()
at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state)
— End of inner exception stack trace —
at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult)
at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)
From here, I went ahead to compare the content index catalog sizes between the 3 mailbox databases in the environment and quickly noticed that the problematic mailbox database had a catalog that was 1.5GB while the others were 9.5GBs or more.
My hunch was that the Veeam backups which take place at 11:00p.m. every evening was conflicting with the index building engine so I proceeded to stop the backups for the evening, forced Exchange to rebuild the index catalog over the evening. After leaving the environment for a day, I went back to test searching on both and Outlook client in Online mode and through OWA and was able to retrieve results older than 1 month.
