When activating a database copy in an Exchange Server 2010 Database Availability Group it may fail with an error message that catalog index files are in a failed state.
——————————————————–
Microsoft Exchange Error
——————————————————–
Cannot activate database copy ‘Activate Database Copy…’.Activate Database Copy…
Failed
Error:
An Active Manager operation failed. Error The database action failed. Error: An error occurred while trying to validate the specified database copy for possible activation. Error: Database copy ‘MB-HO-03’ on server ‘HO-EX2010-MB2.exchangeserverpro.net’ has content index catalog files in the following state: ‘Failed’.. [Database: MB-HO-03, Server: HO-EX2010-MB1.exchangeserverpro.net]An Active Manager operation failed. Error An error occurred while trying to validate the specified database copy for possible activation. Error: Database copy ‘MB-HO-03’ on server ‘HO-EX2010-MB2.exchangeserverpro.net’ has content index catalog files in the following state: ‘Failed’..
When you view the copy status of the mailbox database the content index is in a failed state.
[PS] C:>Get-MailboxDatabaseCopyStatus | fl name, contentindexstate Name : MB-HO-01HO-EX2010-MB1 ContentIndexState : Healthy Name : MB-HO-04HO-EX2010-MB1 ContentIndexState : Healthy Name : MB-HO-03HO-EX2010-MB1 ContentIndexState : Failed Name : MB-HO-02HO-EX2010-MB1 ContentIndexState : Healthy
There are two likely causes of this issue.
- The content index has failed due some error or fault occurring
- Indexing has been disabled for that database (a common configuration for databases hosting journal mailboxes)
Fixing Failed Content Indexes
To resolve the issue update the content index on the server on which it has failed.
[PS] C:\>Update-MailboxDatabaseCopy "MB-HO-03HO-EX2010-MB1" -CatalogOnly
The content index should now be in a healthy state.
[PS] C:>Get-MailboxDatabaseCopyStatus | fl name, contentindexstate Name : MB-HO-01HO-EX2010-MB1 ContentIndexState : Healthy Name : MB-HO-04HO-EX2010-MB1 ContentIndexState : Healthy Name : MB-HO-03HO-EX2010-MB1 ContentIndexState : Healthy Name : MB-HO-02HO-EX2010-MB1 ContentIndexState : Healthy
After the content index is healthy you can attempt to activate the database copy again.
Performing Database Switchovers When Indexing is Disabled
If indexing has been intentionally disabled you can still perform a database switchover using the Exchange Management Shell, by running Move-ActiveMailboxDatabase with the -SkipClientExperienceChecks switch.
[PS] C:>Move-ActiveMailboxDatabase MB-HO-03 -ActivateOnServer HO-EX2010-MB2 -SkipClientExperienceChecks
Assuming no other issues exist with the database the move should complete successfully.
Dear Paul
I have a mailbox database named: Mailbox Database 1174790013- HO009
Previously, because of full disk, this mailbox database failed, Content Index State reported failed
I’ve done enough to get it to mount again as unsuccessful.
Excuse me. I can execute orders
Update-MailboxDatabaseCopy “MB-HO-03HO-EX2010-MB1” -CatalogOnly
so that it can go back to normal?
Thanks
Hi Paul,
We are facing issues in Out Off office auto replay on random user in our organization, On some users its working fine but on some users its behavior quite abnormal some times its not implementing and on the mean time if its works its not reverting. Kindly advice.
Hello All,
Update-MailboxDatabaseCopy “10GB-mailboxes” -CatalogOnly command in Exchange 2016 I got Error :
Database ’10GB-mailboxes\*’ has only one copy. This task is supported only for databases that have more than one copy.
Hi Paul,
Your articles always helpful for me. Thank you.
I need your assistance here, while executing the command ‘Update-MailboxDatabaseCopy “DB111Server” -CatalogOnly
I am getting following error
“A source-side operation failed. Error An error occurred while performing the seed operation. Error: An error occurred while updating the search catalog files from server “ActiveServer” to PassiveServer”. Error: can’t pause the search catalog. Error: Microsoft.Exchange.Search.Common.FteCatalogNotFoundException: SearchCatalog.PauseIndexing failed, error 0x80043629 —> System.ComponentModel.Win32Exception: Unknown error (0x80043629)
—-End of Inner exception stack trace —-
at Microsoft.Exchange.Cluster.Replay.SearchCatalogProxy.pauseIndexing()
at Microsoft.Exchange.Cluster.Replay.CiFilesSeedeInstance.(SeedThreadProcInternal)b___3(Object , EventArgs )
at Microsoft.Exchange.Cluster.Replay.CiFileSeederInstance.REtryCiOperation (Eventhandler evt) .. [Database: DB111, Server: PassiveServer
+CategoryInfo : InvalidOperation: (:) [Update-MailboxDatabaseCopy], CiSeederGenericException
+FullyQualifiedErrorId : FFC07141,Microsoft.Exchange.Management.SystemConfigurationTasks.UpdateDatabaseCopy
Quick reply will be highly appreciated.
Thanks and Regards
Sarfraz
Thanks Paul; this was a huge help on a Monday morning!
Paul,
Thank you very much. This worked well for me. I saw other websites with solutions, but your website is the one the I trust.
Andrew
Hey Paul,
What to do when content index of active and passive mailbox are in crawing state?
It’s very urgent please?
Thanks to read you
Merci
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Wait for them to finish crawling.
Seriously, what would I do without you? …Buy support from Microsoft, probably. THANKS!
Happy New Year to you Paul.
Excellent article. Worked like a charm.
I have one other problem you may be able to help with.
I have moved all of the mailboxes from the offending database but it it still showing that it has a 50Gb .edb file. Is it just a matter of deleting the database seeing as it is probably not going to be used again?
Any thoughts or help would be greatly appreciated.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Moving mailboxes out of a database doesn’t shrink the database file itself. It can only ever grow, not shrink.
If you no longer want the empty database you can remove it via the Exchange console.
First, thanks for the article! Your website is a great help.
Today I discovered that more than half of my (substantial quantity of) database copies showed “failed” for the ContentIndexState. I was able to resolve the issue with the Update-MailboxDatabaseCopy -CatalogOnly command (thanks to talishka for the where ‘Failed’ pipeline command) but I’m a little confused as why this worked. For some of the databases, all copies of the ContentIndex showed failed including the Mounted active database. My understanding is that the update-mailboxdatabasecopy command copies (“seeds”) information from the active database to the non-active copies. If even the active database is failed, how can this information be properly copied and resolve the issue? Shouldn’t I have had to use the ResetSearchIndex.ps1 script to re-index the Active database copy?
Part two: The large list of “failed” contentindexes was quite alarming (presumably this would have prevented automatic takeover in case of server failure!) and I have no idea how long there have been failed contentindexes (have they been building up over time? or did they all suddenly appear?). How can I determine when the failures occurred (particular event IDs in logs?) and what caused them? Is there any way to prevent them? I see you said above in comments you’re not sure what causes them but are there best practices that might reduce their occurrence?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
1) Yes an index reset/rebuild would be necessary if all copies of the database had failed indexes.
2) Unhealthy content indexes will not prevent failover. They are a factor in the best copy selection process, so a healthy index would be preferred over an unhealthy one, but if they’re all unhealthy one of them should mount (assuming no other issues prevent it).
Monitor your server/database health to avoid surprises like this.
STUN-Wah advice Paul, thanks very much, I used your Update command in EMS and the catalog reseeded perfectly on the sick database, all well again, I appreciate your expertise.
Hi Sir,
Please help me, All my database copy in a DAG has a failedandsuspend index content status and when I run Update-MailboxDatabaseCopy -Identity “DB 01MBX101” -catalogonly I get this errors:
WARNING: Seeding of content index catalog for database ‘DB 01’ failed. Please verify that the Microsoft Search
(Exchange) and the Host Controller service for Exchange services are running and try the operation again. Error: The
seeding operation failed. Error: An error occurred while performing the seed operation. Error: An error occurred while
updating the search catalog files from server ‘MBX1’ to ‘MBX2’. Error: A
transient exception from Exchange Search was encountered. Error: -1..
Hope you can help me.
Thanks,
Arnel
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
What have you tried so far?
Hi Sir Paul,
I really need your help when i try to activate or switch my database copy in the DAG I get the following error:
The Microsoft Exchange Replication service does not appear to be running on “xxxmb101”. Make sure that the server is operating, and that the services can be queried remotely.
Please help me.
Thanks Sir Paul,
Happy Holidays!
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
What have you done so far to troubleshoot it?
It helps me. Thanks.
Incredible to see how MS does nothing to address this little problem – the work on the overnext problem … erm… Exchange version seems to be more important than to auto-fix problems that are obvious; be it in the ource or by auto-healing.
Sad to see that a company which claims to be powerful is so weak – but to be honest, I did not expect more…
Rob
Excellent. reseeding the Catalog only worked like a charm. Thank you!!
WORKED LIKE A CHARM…BIG THANKS.
Thanks!!!
Hi Paul,
Have you seen any issues with Exchange 2010 SP3 in EMC >> Server Config >> Mailbox. Under Database Copies > Copy Status not updating correctly with a refresh.
I end up logging into another Exchange server just to see the current correct copy status.
Thanks,
Jeff
Hi Paul,
Really it is very useful.
I have one question , Can you explain how does replication works in DAG, if suppose if any log file got corrupted in passive copy in DAG , how identify and how will resolve it.
Highly appreciated your quick response.
Thanks,
Upendra.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
There is a lot of great information on TechNet that explains how Exchange database replication works.
Hi Paul
I have content index state on one DB as crawling for both the active and the passive noe. Any help would be appreciated
Thank you !
Have a wonderful weekend !
Greetings from Lisboa !
Pingback: Exchange 2010 – DAG server with content index status failed | swissiws
Well, you can use Long Path Tool, it works good for such problems……….
Pingback: Exciting things I have seen on the internet November 18 2013 | 503 5.0.0 polite people say HELO
Pingback: Exchange 2010 Failed Content Index | Fix My Exchange MonkeyBoy
Hi Paul
It’s one of the best Website which I used forever.
Thank you for your short and good guides to resolve problems
Best Regards
Hi Paul,
thanks for this post. I wonder what’s the solution when there’s only one server i.e. only one database.
I noticed index state was failed, I reseed search index but perf mon crawler is telling me there is one mailbox to be seeded. This lasts from the 1st of November. Also, I have error id 4087:
Failed to move active database ‘Mailbox Database DB1’ from server ‘ExSrv1’. Move comment: Database failover initiated by Responder ‘SearchLocalCopyStatusEscalate’ target resource ‘Mailbox Database DB1’
Error: An Active Manager operation failed. Error: The database action failed. Error: This operation is not supported on standalone mailbox servers.
Could you tell me what’s going on? Will crawling process stop? If not, how could I find the mailbox?
Best regards,
Ciprian
Hi Paul, Happy New Year! Finally solved issue, please delete this comment.
Best regards,
Ciprian
Hi Ciprian – can you tell me what the fix was?
Thanks in advance.
Tarek, hello,
Unfortunately I didn’t find out what was the mailbox that caused the trouble so I started out and created a new database where I moved some of mailboxes (I took reverse the size they had). No joy after rendex. I created a new one and moved another ones. At the third one I noticed that after reindex databases were healthy. So, I moved back users mailboxes and delete the temporary databases.
I don’t have a lot of users and a had also a lot of space available – take care that moving a mailbox to another database does not free the space.
I managed to find out the maibox looking the migration logs.
Ciprian
I have opened up a powershell command window as administrator, I have then typed get-mailboxserver |Get-MailboxDatabaseCopyStatus | fl name, contentindexstate and I get the following error
get-mailboxserver : The term ‘get-mailboxserver’ is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At line:1 char:1
+ get-mailboxserver |Get-MailboxDatabaseCopyStatus | fl name, contentindexstate
+ ~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (get-mailboxserver:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
I am just learning Powershell so why wont it work?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Open the Exchange Management Shell (which is PowerShell with the Exchange snapin automatically loaded for you).
Hi Paul,
Thanks for always being a source of help & support for all of us.
Sulex
HI Paul
it’s worked, thanks for your gr8 support.
BR!
Thank you!
Pingback: Exchange 2010 – DAG – Erro ao ativar cópia database. Content Index Catalog Files in Failed State – Windows Brasil
Pingback: Switchover active server in Exchange 2010 DAG failsBilligPC.dk | BilligPC.dk
Worked like a charm!
Thanks!
Worked like a charm. Thank you Paul for this and other helpful information you provide.
Pingback: Exchange 2010 – How to move edb file (Move-DatabasePath) | FICILITY.NET
Thanks for that. Concise and very helpful.
Hi,
I have a Database which has a Catalog Index = FAILED
The database IS NOT part of a DAG. It is a stand alone DB which is use for archiving emails (archiving function in Exchange 2010)
The DB is currently in a DISMOUNT state because of the CATALOG INDEX FAILED & cannot be mount.
I tried applying a command such as:
Update-MailboxDatabaseCopy “Mailbox Database 01EX1” -CatalogOnly
It does not work as the DB is not in a DAG.
I have also try the following command:
[PS] D:Program FilesMicrosoftExchange ServerV14Scripts>.Troubleshoot-CI.ps1 -Database “Mailbox Database 01EX1” –Action DetectAndResolve
However, I did not have any luck with this one either.
How can i correct this catalog index & hence be able to mount the DB ?
Thank You in advance
Ian Salgado
It works! But before typing this commands I needed to restart my Exchange services. Thank you very much!!!
Thank you for that article – it helps me in a short way 🙂
Pingback: Exchange Server 2010 – Failover & Maintenance | f i c i l i t y .net
When ran reset index for mailbox databases, seems this error
Stop-Service : Service ‘Microsoft Exchange Search Indexer (MSExchangeSearch)’ cannot be stopped due to the following error: Cannot open MSExchangeSearch service on computer ‘.’.
At C:Program FilesMicrosoftExchange ServerV14ScriptsResetSearchIndex.ps1:105 char:13
+ stop-service <<<< MSExchangeSearch
+ CategoryInfo : CloseError: (System.ServiceProcess.ServiceController:ServiceController) [Stop-Service],
ServiceCommandException
+ FullyQualifiedErrorId : CouldNotStopService,Microsoft.PowerShell.Commands.StopServiceCommand
I guess search service not respoding for reset,
Any idea ?
Thanks,
Amit
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Is it running when you try to reset it, or already stopped?
You need to run Powershell as an Admin Account otherwise you get this error.
(right click on powershell, choose ‘run as administrator’)
That’s all.
in that case, you need to go to the Exchange script folder where you have Active copy of DB with contect index Failed . you may find it is in default location which is C:Program FilesMicrosoftExchange ServerV14scripts
after going in to Script directory through EMS, run below command
.ResetSearchIndex.ps1
It will ask you to press enter for continue. so say YES.
Once it done, intial state of Contectindexing will show as crawling. later it will turn in to Healthy state.
Cheers!
Hi,
Could you have details on contentindex state failed on active node database server, As of we reseed only passive copy with active node. In case its active node database server how it possible to make its contentindex state failed to healthy
does any one have answer ?
Thanks,
Amit
if there are more than 1 mailbox server in to Exchange environment then you need to run following commnad to know actually which copy of the databse having content indexing issue.
get-mailboxserver |Get-MailboxDatabaseCopyStatus | fl name, contentindexstate
Based on the result, it will give us that which databse on which server is having status fail of content indexing
Hey,
Did you have information on failed index state as error message ” The Time stamp of the Catalog is too old for database”
Please us know,
Thanks,
Amit
Worked for me after changing my database paths and the copy wouldnt become active. Thanks – really great to have helpful people in the commuinity.
Thanks Paul. Worked great for me.
Get-MailboxDatabaseCopyStatus | where {$_.contentindexstate -eq ‘Failed’} | Update-MailboxDatabaseCopy -CatalogOnly
Paul’s command failed for me because “[database]* matched multiple entries” but this sorted it thank you.
Hi Paul,
This works!! Thanks a lot.. Your posts has been a big help to the exchange community..
Hey Paul,
Thanks for this info, it has worked both times I have had this issue. One question I have that I haven’t been able to find anywhere is what causes this to happen in the first place and how can I avoid it?
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
Not entirely sure, to be honest. At least I can’t remember off the top of my head right now.
Noone is very sure about why that happens. We’ve seen it on nearly empty databases more often than on filled ones (for God’s sake!), and see it happen when the GC is not available in a Failover Cluster.
Greetings,
Rob
I found two other articles concerning the same problem, with slightly different solutions, which sounded promising…
http://msexchangeguru.com/2010/12/01/cannot-activate-database-copy-content-index-catalog-files-in-failed-state/
http://blogs.msdn.com/b/pepeedu/archive/2010/12/14/exchange-2010-database-copy-on-server-has-content-index-catalog-files-in-the-following-state-failed.aspx
Unfortunately, neither of these could resolve my problem. In the end, I had to just stop the Microsoft Exchange Search Indexer service on both nodes, then delete the Catalog folder, and restart the service. I just hope this doesn’t happen too often.
Mike
This isn’t working for me. The command completes without errors, but the state of the database doesn’t go back to healthy, and the server keeps logging ExchangeStoreDB 123 events (which says to run the Update-MailboxDatabaseCopy command to reseed the catalog). This has affected two databases in the past two days. The first one was a new database, so I fixed it by deleting and recreating it, but that’s not a long-term solution. Any other ideas? I’ll keep searching. Thanks.
Hi paul, works like a charm!! thnx for the good solution. +1
Regards,
Leon
Hi,
Thanks for detailing that out; however my databases are stuck in a permanent ‘crawling’ state and cannot run the -catalogonly command.
Any suggestions would be greatly appreciated.
The Real Person!
Author Paul Cunningham acts as a real person and passed all tests against spambots. Anti-Spam by CleanTalk.
How permanent? Is it just that your DB’s are big so its taking a while? Does dismount/mount clear it? Or cycling the search indexer service?
Thanks Paul, I’ve seen this before but it’s nice when the error and solution match right up.
It’s work~ Thanks for your help.
We have restarted Microsoft Exchange Search Indexer service. After restarting service all mailbox content indexes are Healthy.
This worked for me!
Only restarted SearchIndexer!!!
Worked for me! Thanks…
Only restart service… and work fine!!! thx!!!