Create SharePoint 2010/2013 Search Content Sources – one script to rule them all

14 Jul
July 14, 2014

In this article I will show you how to create SharePoint 2010/2013 Search Content Sources with a handy powershell script and why you should care.

This is my 100th blog post and therefore it has to be something with SharePoint Search – something good. I had the idea for the script on my mind for quite a while now, but there was no project and no time to create it – until now.

The Problem

During my consultant work I see quite a lot of different SharePoint environments and 90% of the Search Service Applications look like this before I start working there:

2014-07-14 19_52_27-Search Service Application_ Manage Content Sources - Internet Explorer

What is the problem you may ask? I see quite a few. Most intranets I see are rather big and have different sections/departments/regions within their portal – with different requirements of course. Some aggregate their content via Search (aka Search driven applications) and some don’t – some upload documents, some only use SharePoint as archive – as flexible as you can use SharePoint and as different as the requirements can be you should adjust your Search Content Sources accordingly – because everything you configure there will be in the Search Index – everything else wont.

Everything you configure in the SharePoint Content Sources will be in the search index – everything else won’t!

So what is the problem with one content source?

You are not flexible – no different crawl schedules, no priorities – just one setup to cover everything. How about different settings for DEV / QA systems ? How about different crawl schedules for people search? LOB / BCS, external systems? Or people complain that some results appear too late in the search or that they are not there at all? Read on!

The Solution

You can create content sources with PowerShell – that’s a good thing and enables us to automate it. So the easiest way to create a Content Source would be (except creating them by hand in the central admin) with this one-liner:

You will be asked for the name of the Search Service Application and then it creates the Content Source for you  – good, simple and works. But apparently no crawl schedule. And imagine creating that for 16 different content sources with different crawl schedules and what not. And maintainable and readable it should be, too. So we need a more sophisticated script for that with reproducible results – in other words we need a xml config file and a powershell with the logic.

My script consists out of two files – one is the actual powershell with all the logic and the second is a xml config file with all the parameters – lets have a look inside the xml file.

XML Config File

The config file contains every to be created/updated content source

The config file contains every to be created/updated content source

In the 3rd line you have to specify the Search Service Application in my case this is “Search Service Application”.

In line 13, 46 and 81 I configure three different content sources – if you want to create more, then you only have to copy one block and maybe for the ease of use the comments surrounding the block starting with <ContentSource> and ending with </ContentSource>.

Every <ContentSource> block has a unique name, a type (currently tested is SharePoint and File) and a crawl behavior (CrawlSites = crawl the site collection; CrawlVirtualServers = crawl the entire host/web app)

In every <ContentSource> you can define the start addresses of the content source within the <Url> element. If you have multiple addresses, separate them with a new line or return.

Then every <ContentSource> can have up two crawl schedules. The comments above and the three provided examples should give you a good understanding how to configure the crawl schedules.

The script

You don’t have to read the script or customize it – everything is configured in the xml config – the script just reads the xml file that must be in the same directory. The script has no parameter but must be run in an elevated PowerShell on a SharePoint Server.

Run the script in an admin powershell

Run the script in an admin powershell

Running the script with the provided xml would give you the following output:

... and the same as screenshot!

… and the same as screenshot!

This will result in the following configuration in the central admin:

3 new content sources

3 new content sources

one example with the urls set

one example with the urls set

... and the crawl schedules!

… and the crawl schedules!

Important: This script does not remove or rename content sources (it simple can’t detect those changes)! If you want to rename an existing content source you can either delete the content source in the central admin or rename it there and in the xml file.

Important: Please keep in mind that deleting a content source or changes source addresses within a content source deletes items within your index! There will be an automatic cleanup of the index once you remove urls from a content source and you have to recrawl the items if you still need them!

If you change the parameters in the script, e.g. the crawl schedule and run the script again – the script will update the content source for you. So if you have a different environment with the same configuration, just copy both files – if they are different, you have to adjust the config file accordingly.

How to get the script

I released the script based on the MIT license in this GitHub repository:

https://github.com/MaxMelcher/SPContentSource

Feedback / next steps

Some parts of the script are in a very rough state – I will update the script very soon because I need it in a project. Some of the next steps are:

  1. Handle priorities
  2. Support continuous crawl (SP2013 only)
  3. Implement BCS and other types
  4. Add some more logging
  5. Add some error handling
  6. Add crawl rules

If you encounter any bug or problem please drop me a line or open an issue here. Contributions and pull requests or any other feedback is much appreciated.

What happened to ‘the’ ULSViewer and how to still get it?

04 Jun
June 4, 2014

MSDN Archive has been retired this week – I do not know why they retired it before there is an adequate replacement for the majority of the tools, but it has a huge impact on my work. One of my favorite tool, the famous ULSViewer is was hosted there.

You will be greeted with this beautiful page when you try to download the ULSViewer

You will be greeted with this beautiful page when you try to download the ULSViewer – no alternative download location so far!

ULSViewer – why would you miss it?

Why I love the tool?

  1. its tiny – with just 487kb you can grab it, copy it over the slowest RDP connection and start working in less than a minute.
  2. its super fast – even with the biggest ULS Logs you can still filter and search for correlations, full text and what not.
  3. its just one file – no installation, plain .exe file, it can not be simpler.
  4. the filtering and coloring is superb – reading the uls can be very challenging (sometimes you need true SharePoint Ninja Skills), but the filtering and coloring really helps me a lot.
  5. it has real-time support – watching what happens right when it happens – I hate wait times.
  6. it supports all versions – support for SharePoint 2007 – 2013 is just great!
  7. its free – I  used it for roughly 30 clients – no licensing costs helps quite a lot.

Additionally, just yesterday I solved an issue with the ULSViewer where the coloring and the real-time functionally was a major part of the issue identification.

Formatting, highlighting and real-time support

Formatting, highlighting and real-time support helped me solving an authentication issue

BringBackULSViewer.com

Trevor Seward (@NaupliusTrevor) launched a page to express that we want the ULSViewer back – most favorably released with an Open Source license hosted on Github or Codeplex.

Bring Back ULSViewer! - Internet Explorer_2014-06-04_10-30-57

If you did not participate in the survey – please do so now! Collecting our voices helps!

30 Seconds Survey

30 Seconds Survey

How to still get it?

There are three options:

1. You downloaded it already

Easy one – just don’t loose the file. OneDrive it and you cant loose it, right ?

2. You can download it from Todd Klindt

If Tood uses ULSViewer, it has to be good, right? Todd mirrors the ULSViewer on his blog – download it here.

3. Coolest solution: Download it with PowerShell

Install chocolatey (takes 30 seconds) and use the following command

This downloads the ulsviewer and creates a shortcut on your desktop – in less than a minute in total – quite handy and super fast!

Clipboard_2014-06-04_10-50-24

_2014-06-04_10-53-23

More chocolatey?

  1. Install SharePoint Designer with Chocolatey
  2. Updated Chocolatey with more tools
  3. Little helpers for “lazy” developers
  4. Install ULSViewer with Powershell

Installing SharePoint 2013 SP1 with SQL Server 2014 on Windows Server 2012 R2 – Picture Love Story

30 May
May 30, 2014

I am trying something new for me: This will be a huge picture post.

Today we have a bridge day in Germany and that means most of my colleagues and clients are taking the day off. Ideal time for me to replace my SharePoint installation and bring it to a new level. Why did I upgrade? Well, lets say I have a fable to try everything new out before customers asks me some questions about it –  and I kinda like shiney new stuff. Hands on, here we go.

Requirements

In this post I show you how to install a small SharePoint 2013 Development Environment in 49 pictures – and a picture is worth a thousand words, right?

You need

  • a virtualization environment with a preconfigured Active Directory (Azure, Hyper-V, VMWare –  I show it with Hyper-V on Windows 8.1 Update 1)
  • the installation images, I got mine from my activeMSDN subscription:
    • Windows Server 2012 R2 with Update (en_windows_server_2012_r2_with_update_x64_dvd_4065220.iso)
    • SQL Server 2014 (en_sql_server_2014_enterprise_edition_x64_dvd_3932700.iso)
    • SharePoint 2013 SP1 (en_sharepoint_server_2013_with_sp1_x64_dvd_3823428)
    • Visual Studio 2013 with Update 2 (en_visual_studio_premium_2013_with_update_2_x86_dvd_4238022)
    • AutoSPInstaller (version 3.96)
  • coffee & internet or some work during the installation.

How to “read” this post?

In this post I intentionally leave a lot out – most of this the “next next next” dialogs during setups or what I consider basic stuff. Sometimes I do not show how to get to a specific dialog – if you are stuck, please drop me a comment.

When you open one picture, you can jump to the next one if you just click on the image. – the description is above it.

 

Let’ start: Windows Server 2012 R2

1. Create a new VM - give it a cryptic name so only you understand whats in the box :)

1. Create a new VM – give it a cryptic name so only you understand whats in the box :)

2. Generation 2 has way shorter boot times and is more performant

2. Generation 2 has way shorter boot times and is more performant

3. Give it as much RAM as you can - I will change it later once I shut down my work VM.

3. Give it as much RAM as you can – I will change it later once I shut down my work VM.

4. You need some network to install those precious windows updates and tools later on

4. You need some network to install those precious windows updates and tools later on

5. Storage: 127GB is good enough, you could extend later on.

5. Storage: 127GB is good enough, you could extend later on.

6. Pre-mount the Windows Server 2012 R2 image

6. Pre-mount the Windows Server 2012 R2 image

7. Ready, steady, go! Start the machine.

7. Ready, steady, go! Start the machine.

8. Thats new with Generation 2 VMs - Full Screen, copy & paste transfer and much more.

8. Thats new with Generation 2 VMs – Full Screen, copy & paste transfer and much more.

9. Select your language options - I always install English with German keyboard layout so I dont need to break my fingers when I type @ or \

9. Select your language options – I always install English with German keyboard layout so I dont need to break my fingers when I type @ or \

10. Time for the first coffee

10. Time for the first coffee

11. I always use pass@word1 as password - don't tell anyone!

11. I always use pass@word1 as password – don’t tell anyone!

12. Network settings - I disable IPv6 because it complicates things for me.

12. Network settings – I disable IPv6 because it complicates things for me.

13. With this IP my Internet Connection Sharing does always work - DNS is my Active Directory Server.

13. With this IP my Internet Connection Sharing does always work – DNS is my Active Directory Server.

14. Time for the domain join and a decent name. Lets call the server sp2013-2.

14. Time for the domain join and a decent name. Lets call the server sp2013-2.

16. Lets try to join my domain "demo.com"

16. Lets try to join my domain “demo.com”

17. A prompt is always a good sign here. Enter Domain Admin credentials.

17. A prompt is always a good sign here. Enter Domain Admin credentials.

18. Yes! It worked.

18. Yes! It worked.

19. Important: Configure Windows Udpates. Because I frequently present, I dont want the updates to be downloaded or installed when I dont want it. Cell usage is quite costly, please ask dear windows!

19. Important: Configure Windows Udpates. Because I frequently present, I dont want the updates to be downloaded or installed when I dont want it. Cell usage is quite costly, please ask dear windows!

Install SQL Server 2014

20: Time for SQL Server 2014

20: Time for SQL Server 2014

21: Select what you need - for me only 3 checks. Database Engine is obvious.

21: Select what you need – for me only 3 checks. Database Engine is obvious.

22: Management Studio is important, too.

22: Management Studio is important, too.

23. Lets install - next next next, oh dear!

23. Lets install – next next next, oh dear!

24. Open server manager, top right go to "Add Roles and Features"

24. Open server manager, top right go to “Add Roles and Features”

25. Add .NET 3.5 and hit next

25. Add .NET 3.5 and hit next

26. I always add a named instance, lets call it SharePoint.

26. I always add a named instance, lets call it SharePoint.

27. Add some users with admin privileges.

27. Add some users with admin privileges.

28. Press next and wait until this screen appears. Coffee?

28. Press next and wait until this screen appears. Coffee?

Installing SharePoint 2013 SP1

Please note: you need bundled SP1 to install on Windows Server 2012 R2.

29. Finally - the base is set, lets install SharePoint 2013 SP1.

29. Finally – the base is set, lets install SharePoint 2013 SP1.

30. I install the PreReqs manually with the GUI - am I old school or what?

30. I install the PreReqs manually with the GUI – am I old school or what?

31. Then I configure my AutoSPInstaller File with everything I need: From Search to BCS - with a single WebApp that contains everything (Best Practice)

31. Then I configure my AutoSPInstaller File with everything I need: From Search to BCS – with a single WebApp that contains everything (Best Practice)

You can grab my config file here: AutoSPInstallerInput.xml (just rename the extension and drop it to the AutoSPInstaller SP Folder)

32. Copy the binaries to the SharePoint 2013 folder inside the AutoSPInstaller dir

32. Copy the binaries to the SharePoint 2013 folder inside the AutoSPInstaller dir

33. The farm user needs some rights - my user is called spfarm

33. The farm user needs some rights – my user is called spfarm

34. Give it DB Creator and Security Admin rights.

34. Give it DB Creator and Security Admin rights.

35. Hit play or better start the magic AutoSPInstallerLaunch.bat

35. Hit play or better start the magic AutoSPInstallerLaunch.bat

36. Green lines are good lines

36. Green lines are good lines

37. At least no red lines, right?

37. At least no red lines, right?

38. A familiar window

38. A familiar window

39. Some progress - coffee?

39. Some progress – coffee?

40. Green green green.

40. Green green green.

41. Creating web apps, service applications, site collections, configuration - this tool is soooo impressive.

41. Creating web apps, service applications, site collections, configuration – this tool is soooo impressive.

42. Voila - Central Admin opens, no red health issue thingy.

42. Voila – Central Admin opens, no red health issue thingy.

Install Visual Studio 2013 with Update 2

43. Installing Visual Studio is so easy, I skip the screenshots. Hit next next next.

43. Installing Visual Studio is so easy, I skip the screenshots. Hit next next next.

44. There we go.

44. There we go.

Install the tools

45. Installing 10 SharePoint must have tools - chocolatey is the answer!

45. Installing 10 SharePoint must have tools – chocolatey is the answer!

Go to http://chocolatey.org to get the package manager to install other tools. Read this post for more info. Then open a new PowerShell and copy this into it:

cinst 7zip cksdev11 dotPeek gitextensions GoogleChrome notepadplusplus SearchQueryTool SharePoint.HiveShortcut.Desktop SharePoint.HiveShortcut.Explorer SharePointManager2013 SPCAF SPSF ulsviewer cksdev2012

46: Chocolatey - sweet!

46: Chocolatey – sweet!

47: Go go go!

47: Go go go!

48: Green lines indicate new SharePoint Tools installed.

48: Green lines indicate new SharePoint Tools installed.

49: Voila, tools installed - actual work can start now!

49: Voila, tools installed – actual work can start now!

Closing thoughts

Well, writing and preparing this blog post took longer than the actual installation – AutoSPInstaller did a decent job, installing my “baseline” SharePoint Toolbox with chocolatey is easy – welcome brave new world! If I could focus on this 100% full time, I think I can install everything in roughly 2 hours, not bad, right?

To be honest, not everything worked like a charm during this setup – how could it? I did not prepare a thing: I had to run autospinstaller like 10 times to install SharePoint – but everytime I forgot a value and the tool was so nice to tell me what was missing – but you can always restart the tool and it continues, awesome! Additionally some of the chocolatey packages failed during install – I will try to fix some of them (you cant complain cause the tools are free, right?). There is room for improvement here.

But to close this post, installing everything from scratch was good fun!

Do you like the style of the picture post? If so, please leave a comment!

ShareCamp 2014 Recap – SharePoint all weekend long!

30 May
May 30, 2014

ShareCamp 2014 was last weekend (05/23/2014 – 05/25/2014) hosted by Microsoft in Munich and it was the best community event I attended this year – and I repeat that now for 3 years in a row. So why is it that great? The answer is simple but unexpected: Its organized by and depends on active participation of the community: Each attendee can present their topics and questions in a session – if the other attendees wanna hear their stuff. Great concept (see barcamp or unconference for more background), I like it. Why? Because its not the 100% well prepared, clinical presentations – its from guys like you and me, you can have fun during the session, interrupt the speaker with questions and more. Just great and very entertaining, the hours passed so fast.

20140525_083121000_iOS

Session board with high-quality sessions and innovative topics

20140525_083126000_iOS

6 sessions in parallel – normally you would have to pay for that.

So how does it work?

Every morning all attendees gather and do the session voting. So if you want to present or discuss something, you tell us your session title and a brief abstract – then the attendees vote and if you are lucky (its easy to get votes imho) you get a slot and a room to have some fun.

Sessions I attended

I was a little late on Saturday (and apparently on Sunday, too) so I did not attend the session votings at all this time – but I attended 6 or 7 sessions, some of them I will describe below:

First session was by Sebastians (@hupseb) session about Dev Environments hosted on Azure – solid talk, even though that the demo did not work because of poor wifi.

Slides: http://melcher.it/s/2D

20140524_120422582_iOS

Search-driven Intranet by Sascha Henning (@sascha_henning) was very interesting, too. Whole Intranet was powered by search – and as you know, I <3 Search.

Some ideas I will steal copy, I totally liked the “daily menu app” aka Word Document rendered with Office Web Apps.

Picture by @h_ulbricht

Office 365 Development Experience Exchange by Christian Gross was very entertaining. The attendees did not have that many experiences with O365 dev, but Christian told us how much trouble his company had to migrate their on-prem tool to the cloud. We discussed what problems we see with the rather unstable API of O365 and the problems with the 2-weekly releases. Good stuff to keep in mind.

Easy site provisioning with Collaboration Manager again by Christian Gross – this was a product session and good to see how powerful this 3rd party tool is. You can create team site templates within SharePoint. Then you can provision them based on this template. The most impressive part was to see the synchronization when the template changed – you ever tried to update sites after the site definition/template changed? There is a tool for that.

Location/Context-aware apps by Christian Heindel: Some good ideas about context/location aware apps in this session. How about promoting the time tracking apps on Friday when people track their hours?

 

Christian Heindel about context-aware apps

Christian Heindel about context-aware apps

Sessions I presented

My plan was to present a session about Search-driven apps with SharePoint 2013 – but because I was in the hospital in the morning (long story!) I could not present my session during the session vote. But luckily Nicki Borell (SharePoint “Search” MVP) had an “open” search session and asked me to co-present. So we both faced around 30 people with a multitude of SharePoint questions from Managed Properties, Search Architecture, Search Tools (I presented the Search Query Tool v2) , BCS search connectors and future improvements of search – in just one hour! I hope the attendees had as much fun as we had.

So here are my slides (I did not present all of them): http://melcher.it/s/2E

The second session is an evergreen – I think I presented this one the 3rd or 4th time with updated slides of course: SharePoint Best Tools or SharePoint Toolbox. I am always astonished how many people develop or administer SharePoint systems without proper tools – and there are soooooo many tools around to make our lives easier. To name some: CKSDEV, SharePoint Manager, AutoSPInstaller, SharePoint Search Query Tool (<= my favorite), ULSViewer, Feature Admin, SharePoint Manager, Balsamiq, SPCAF, …

SharePoint Tools – here I show the awesome chocolatey “meta” tool. Picture by @h_ulbricht.

More in my slides: http://melcher.it/s/2F

More Pictures

Whats up next?

A conference needs a raffle, right?

200 attendees group picture directey by Michael Greth (@mysharepoint)

20140525_112844430_iOS

Exhibit hall

ShareCamp - born2share._2014-05-30_16-26-52

Sponsored by those great companies – thanks!

Closing thoughts

Well, there was one thing that I totally did not like this year. ShareCamp is now so successful that its hard to get a ticket. There are around 200 available tickets and they were gone in roughly 4 hours!!! So if you forget to register just one day – you were most likely out of luck and on the waiting list. This leaves the taste of an “insider event” but it should be for the community – and as it seems there is a huge demand in Germany. So why not scale up next year?

Additionally I would love to have 2 hour sessions slots – some well chosen topics (voting upfront like the SP24 conference did it) for special topics (search? app development? Just something with a broader range and 1 hour is just not enough).

That’s it for this year – thanks again to all organizers, MVPs, sponsors and Microsoft for this great event! I had a blast.

It’s not too late! Attend SP24 – The free 24h worldwide SharePoint conference!

16 Apr
April 16, 2014

This is a last minute call for SP24 – The free 24h worldwide SharePoint conference – the free 24h worldwide SharePoint conference starts today/tomorrow and I am presenting!

STARTING TODAY

Register for FREE now (it is not too late) and skip lunch, breakfast – or what every you are doing and be part of the great community event! 1 month ago you had to pay at least $1,500.00 to see some of the speakers – again SP24 is free!

FACTS FACTS FACTS!

  • SP24 lasts for 24 hours and will follow the sun from New Zealand to Hawaii.  It will begin on April, 16th 2014 at 10 pm GMT and will end 24 hours later.
  • SP24 will feature speakers that are voted for by the community (Note: some slots are already reserved for experienced speakers and world-renowned SharePoint experts) – you can see the agenda here.
  • SP24 sessions will be hosted live and you will have the possibility to interact with the speaker and other attendees through integrated chat.
  • SP24 has 2 tracks, one to appeal to Business Users and one for Technical Users such as developers and administrators.
  • SP24 is going to be held entirely on-line.
  • SP24 is completely FREE!

My Session: Search-First Migration for SharePoint 2013 – Benefits / Disadvantages

I was lucky enough to grab a session – you can watch the teaser here for it:

Official Start:
GMT time: 17.04.2014 11:00 AM GMT
German time: 17.04.2014 01:00 PM CEST

I will be in the chat during my session and answer question!

So, who stays up the whole time?