He’s dead, Jim: SharePoint 2013 Search Troubleshooting

13 Oct 2014
October 13, 2014

Recently we had some strange search issues in our very own intranet and its time to blog about some SharePoint 2013 Search Troubleshooting.

At first I noticed that our Search results were quite old, usually that is not the case because we enabled continuous crawling and I was surprised that it worked that well. How I noticed? I have tiny app that utilized search and supports our core business process: billing. But it suddenly stopped finding new items.

Issue 1: Broken web.config on our search server

As mentioned above I noticed that new items in our SharePoint Environment did not show up – you can quickly confirm that when you go to your search center and search for *. Then check the Last Modified Refiner and as you can see below: we had a problem.

image

Last Modified refiner – with zero results for today

But once you identified the problem you can tackle it. In this case I told our internal IT that there is something wrong, we checked the crawl log: ~560.000 errors.

That this was not monitored is a different story – but the error I saw that was quite familiar: “Search error: crawler could not connect to the repository”. That means either the SharePoint is not running, there is a firewall in between – or as in our case it returned HTTP 500 errors for every request. I could easily confirm that by opening a browser on the SharePoint crawler itself and open the target site. I looked at our IT admin: “Did you change something?” –  he: “We deployed a feature and that did not work out that well…” – me: “Did you remove it everywhere except the search crawler?” – he nodded. The in this case broken web.config was quickly restored – started a full crawl to get rid of the many errors and we should be fine, right? Right?!!

Issue 2: Cannot plan query for index system

Because full crawls tend to take quite a while – even with like 100 dp/s – I checked the search results the next morning:

Search

That did not work – every search query resulted in that error. The important part is the correlation ID

Ouch – what happened? I right away called our admin: “What did you do?” – he said  “I just changed the password of the farm account via central admin like I did several times” – mhh, that should work, nothing to blame here. So I asked him to send me the uls logs and some screens of the SharePoint Search Service Application:

image

Full crawl seems to work – a bit slower than it should tho

Crawl logs do not look too suspiciously.

image

Event log #1

image

Huston – we have a problem!

Event log – useful as ever to identify that you have a problem but sometimes it does not help you identify what problem you have… so I was keen to see the ULS entry for the correlation ID I saw during the search – so I got it:

SearchServiceApplication::Execute–Exception: Microsoft.SharePoint.SPException: Tried IMS endpoints for operation Execute: Cannot plan query for index system SP2f28f6df0f54. Index fragment ‘1’ has no available cells. Cell statuses: [Cell I.0.1 on node IndexComponent3: Cell status is set to ‘not available’ (cell out of sync or seeding)] at Microsoft.Office.Server.Search.Query.Ims.LoadBalancer.RoundRobinLoadBalancerContext.NextEndpoint(String operationName, String failMessage) at Microsoft.Office.Server.Search.Administration.SearchServiceApplication._ImsQueryInternalType.DoSpLoadBalancedImsOp[T](ImsBackedOperation`1 imsCall, Int32 timeoutInMilliseconds, Int32 wcfTimeoutInMilliseconds, String operationName) at Microsoft.Office.Server.Search.Administration.SearchServiceApplication…. fe0fc09c-8c99-b017-25c9-d3feac78a369

That sounds not that healthy, especially the Cell status is set to ‘not available’ (cell out of sync or seeding). The index is made out of cells – if they are not there, Search is very unhappy. I can only conclude that changing the password during a full crawl is a bad idea.

So we first tried to restart the SharePoint Search Service – no result.
We restarted the entire box – no result.

So we heavy-heartedly did an index reset and the search results immediately came back – empty but working again. But trust me, resetting the index during a workday, with many search-driven applications around is not the greatest experience you can have – and you shouldn’t. There is room for improvement in our intranet, do you agree?

Max Melcher
Follow me!

Max Melcher

Maximilian Melcher (MCSE, MCPD) is a Principal Consultant working at Alegri International Services in Munich, Germany. Max is a specialist in SharePoint technologies focused on search, social computing, web content management and collaboration. Max has led SharePoint implementations for Dax 30 companies since 2009.
Max’s free time is spent on twitter (@maxmelcher) mostly with a good coffee in his hands.
Max Melcher
Follow me!
Tags: , ,
0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close