| Earlier this week, we’ve had a discussion about the benefits of social media at SharePoint Connections in Bavaria, Germany. One of the benefits was that social media helps to find old friends – looks like my life proves this immediately. I’ve just booked my flight to Boston for this Summer earlier this week – and turned out, one of my best friends will be there, same time. (Funny thing, we live 10 min driving from each other and our children are very good friends too. Will be fun to have a kind of “mothers’ party” in Boston, without our families :)) Last year I’ve seen Andrew Connell there, accidentally. How big is the mathematical chance for seeing a friend in Boston if you live in Europe and he lives in Florida?… Today I was flying to Toronto via Amsterdam. Checked in at Schiphol by Foursquare, and in less than 30 min, Christian Buckley checked in the same airport… He had a layover there heading down to South Africa for SPSJHB… Then arrived in Toronto, and an old friend’s message was waiting for me on Facebook: “hey, I moved to Toronto recently, what about a coffee this afternoon?” – I haven’t seen him for about two years… Literally, I feel the world is much smaller outside in Hungary than inside. Social networks make it so small. I LOVE it, having so much fun when travelling! Anyone in Boston this week’s second half? ;) |
| Recently I’ve been working for a customer where I’ve found some interesting requirements: they had several content sources and wanted to crawl them one by one after each other. Scheduling the incrementals for fix time was not a good solution as their content incrementals were very hectic: incremental crawl for the same content source took 5 min at one time, then 1.5 hours next time. And of course, they didn’t want idle time. But we cannot define these kind of rules from the UI, so the ultimate solution was PowerShell. First, we need to be able to start the crawl. Let’s talk about Incremental Crawl only this time. Here is the PowerShell script for this: $SSA = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application" $ContentSourceName = “My Content Source” $ContentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource -Identity $ContentSourceName $ContentSource.StartIncrementalCrawl() It’s an easy one, isn’t it? Next step is checking the status of this content source. We need this for several reason, for example we want to start the crawl only if it’s in Idle status, or we want to display the current status of the crawl in every minute, etc. Here is the PowerShell command you need: $ContentSource.CrawlStatus What values can it have? Here you are, the list of crawl statuses: - Idle
- CrawlStarting
- CrawlingIncremental / CrawlingFull
- CrawlPausing
- Paused
- CrawlResuming
- CrawlCompleting
- CrawlStopping
Ok, we can decide the status now, we can start a crawl. How to make it “event driven”? Here is the logical sequence we have to follow: - Start the crawl of a content source.
- Wait until it’s done.
- Take the next content source and repeat the steps 1. and 2. until you’re done with each content source.
- Repeat this sequence.
First step is creating a function if we want a nice code. Here you go, my first one: function Crawl
{
#Start crawling
$ContentSourceName = $args[0]
$ContentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource –Identity $ContentSourceName
$CrawlStarted = Get-Date #Check crawl status
if (($ContentSource.CrawlStatus -eq "Idle") -and ($CrawlNumber -eq 0)) {
$ContentSource.StartIncrementalCrawl()
Start-sleep 1
Write-Host $ContentSourceName " - Crawl Starting..." do {
Start-Sleep 60 # Display the crawl status in every 60 seconds
$Now = Get-Date
$Duration = $Now.Subtract($CrawlStarted) # Duration of the current crawl
$Speed = $ContentSource.SuccessCount / $Duration.TotalSeconds # Speed of the current crawl, docs/sec
Write-Host $ContentSourceName " - " $ContentSource.CrawlState (Get-Date).ToString() "-" $ContentSource.SuccessCount"/" $ContentSource.WarningCount"/" $ContentSource.ErrorCount "(" ("{0:N2}" -f $Speed) " doc/sec)"
} while (($ContentSource.CrawlStatus -eq "CrawlStarting" ) -or ($ContentSource.CrawlStatus -eq "CrawlCompleting") -or ($ContentSource.CrawlStatus -eq "CrawlingIncremental") -or ($ContentSource.CrawlStatus -eq "CrawlingFull" )) Write-Host $ContentSourceName " - Crawling Finished"
Write-Host ""
}
}
This is how you can call this function: Crawl("My Content Source") Some additional steps you might need: - If you want to run this script once a day (need daily incrementals only but would like to be done as quick as possible), just schedule this script as a Windows task.
- If you want to run this script during your day only (and release the resources for some other jobs for nights, for example), you can do the “start in the morning and start in the evening” logic. I’ve made a simple example in my blog post a few months ago.
- If you want to run this sequence all day long, you might insert this logic into an infinite loop. (But be careful, sometimes you’ll need to run full crawl and then you have to stop running this script.)
- You can insert some other steps into this script too. If you want to do something (logging, sending some alerts, etc.) when the crawl starts / stops, just do that here. It’ll be your custom “event handler” on the crawl events.
- You can even write the output of this script to a file, so that you’ll have your own “crawl log”.
The scripts above works fine with both SharePoint Search and FAST Search for SharePoint. Enjoy! |
| My very first travel in 2012 just getting to the end (yes, I’ve been home for 3 months now, it’s a kind of unbelievable…): MVP Summit 2012 is over, I’m sitting on the plane back to home. This was my 3rd Summit, being awarded as SharePoint Server MVP for 4 years. And this was the best ever, I can tell you. Of course, the first one was memorable too, being there as a brand new MVP, meeting fellow MVPs for first time, and getting the first personal impressions while shooting them on paintball, and learn that my ‘superstars’ are real persons, and what a persons! It’s a kind if one-time-in-the-life experience. This Summit was different, of course. I’ve seen a LOT of old friends, spent a lot of time with them (although I feel I should have spent much more with some of them). It was great to see everyone again, after a shorter or longer period of time. Also, I’ve met some new guys too, as always. This is what I LOVE in the SharePoint community, it’s like a big family, even if we met face by face occasionally only. The content was also awesome, great sessions and topic. But I cannot tell you any more :) What’s next? I have way too many plans for this year, including: More to come, I have several other sessions submitted but not approved or finalized yet, for example in London, Boston, New York… Busy year, right? Will keep you updated, back to the everydays now… |
| If you read my blog, follow me on Twitter, or sometimes meet me at conferences, SharePoint events, you might know I’ve had a lot of changes in my life in the past 1.5 years. But it’s now public: there is one more, big change you didn’t know so far. This story begins back in the Summer of 2010, when I’ve started to work as an Independent Consultant after leaving the company I was a co-owner and co-founder of. Short afterwards, I joined BA Insight as a part-time subcontractor. It’s been a pleasure to become the part of this great team: I’ve been focusing on Enterprise Search and Content Management for many-many years and here I could really get improved, learn a lot and contribute to exciting projects and products. It’s amazing when you can learn and experience something new every day! When you can contribute to amazing products. When you can work with great people. As months passed away, I had to realize I love to work here more and more, day by day. I’ve been involved not only to more and more cool projects, but also to the work of product teams “behind the scenes”. And finally it turned out it was like an engagement – I’ve been spending more and more time with BA Insight and wasn’t even thinking about my “independent consultant” life. And then, in this January, the “marriage” happened: I joined the company as a full time Senior Solutions Consultant and Project Leader and I have to say: I found what I was working for in the past more than 10 years. “In her role, Molnar will help BA Insight’s enterprise customers craft and deploy search solutions that enhance productivity and reduce costs. Additionally, Molnar will play a pivotal role in BA Insight’s efforts to develop innovative technologies that help organizations leverage the full power of Microsoft SharePoint and FAST Search across the enterprise.” (by BA Insight News and Press Releases) |
| I’ve got a lot of questions about my life, my past and my personality recently, so that I’ve decided to share some stories with you. Not because I’d like to show you how ‘good’ I am but to show you: it’s possible to change your life if you really want it. So, let me start with the years at the university. I’ve made my high school in a small Hungarian town’s small high school, with special classes on Math and Physics. My parents were very poor and the school was very small and not too good, but I had some VERY good teachers there who I will be grateful in my whole life. They were supporting me in my ‘crazy’ computer mania, they let me in to the computer room whenever I wanted, they gave me a lot of books (don’t forget: by that time we hadn’t have Internet yet), etc. I was the very first student in the school who had email address but couldn’t use that as hadn’t have anybody with email address around me yet. I made a lot of cool programs there, including a German-Hungarian dictionary with more than 26,000 words, some e-learning programs for the subjects I didn’t like (history, for example), chat program over a Norton server with one of my friends (who was a computer fan too, of course), etc.
I’ve had a lot of fun there. Then I had to decide what’s next. For me, it was obvious to choose the best university in the country, so that applied to the Budapest University of Technology and Economics (BUTE) and I’ve got accepted immediately. It was one of the biggest wonder in my life, I could leave the small town and moved to Budapest! Wow… But after a few months, my parents realized they couldn’t afford my education. After the biggest wonder the darkest days came.
Then I made the decision to start working instead of leaving the school. I didn’t left my class but started to work as a developer at Graphisoft in parallel.
I have had mixed feelings about this.
First, it was amazing to work for such a great company. At the beginning I had to work with GDL, ArchiCad’s Graphical Description Language – can you imagine what a fun to write a code that finally draws the bridges of Budapest? My favorite one is still the Elisabeth Bridge because of its simply math model and beauty. Then, I had to improve GDL. You know, programming the framework for a program language. Sounds good, right? Yes, it was an amazing job, amazing time and I could learn a lot. But at the same time, you know, I still was a student, had to learn (sometimes should have learn much more…), and yes, as every typical student I wanted to socialize much more. While my friends went out, I was working to be able to stay at the university. While definitely learned a lot, like time management, self-discipline, working hard, etc., it was a hard, very hard time of my life. When .NET came out, I’ve been working for a T-System company as a developer and got the task to ‘explore’ the new language and framework. Then, I’ve got a kind of new ‘research’ task: me and two of my colleagues had to make a .NET forms application that was able to make operations on Exchange public folders. Operations, like Search. With the API of something that was called “Tahoe”… It was my very first meeting with SharePoint. I didn’t know what was this, I didn’t even know it had any UI, the only thing I new there was an API that had Search functions and I had to drive it to be able to work over my Exchange public folders… A few months later, my managers decided to give me a new project. You know, like “you already know SharePoint”… Then one more, one more, one more SharePoint project even after some company changes again, but I always have got bigger and bigger tasks, projects so I had to explore the product more and more. But still, everybody though I knew SharePoint even before I knew about that… By the time I finished my studies as M.Sc. of Computer Sciences I was working as a software architect and had my own SharePoint classes in a training center. I still didn’t know I wanted to focus on SharePoint, but SharePoint wanted me way too much… |
| Refiners are cool either if you use SharePoint Search or FAST, it’s not a question. I very like them, they give so much options and power to the end users. But there’s a very common question around them: the deep and shallow behavior. As you know the definitions very well: FAST Search for SharePoint has deep refiners, that means each result in the result set is processed and used when calculating the refiners. And SharePoint Search uses shallow refiners, where the refiner values are calculated from the first 50 results only. These definitions are easy, right? But let’s think a bit forward, and try to answer the question that pops up at almost every conference: Why some Refiner values are not visible when searching? Moreover: why they’re visible when running Query1 and hidden when running Query2? For example: let’s say you have a lot of documents crawled, and you enter a query where the result set contains many-many items. Thousands, tens of thousands or even more. Let’s say you have some Excel workbook in the result set that might be relevant for you, but this Excel file is not boosted in the result set at all, let’s say the first Excel result is on the 51th position (you have a lot of Word, PowerPoint, PDF, etc. files on the positions 1-50). What happens if you use FAST Search? – As the refiners are deep, each result will be processed, so your Excel workbook. For example, in the Result Type refiner you’ll see all the Word, PowerPoint, PDF file types as well as the Excel. Easy way, you can click on the Excel refiner and you’ll get what you’re looking for immediately. 
But what’s the case if you don’t have FAST Search, only the SharePoint one? – As the first 50 results is processed for the refiner calculation, your Excel workbook won’t be included. This means, the Result Type refiner displays the Word, PowerPoint, PDF refiners but doesn’t display the Excel at all, as your Excel file is not amongst the top results. You’ll see the Result Type refiner as if it there wasn’t any Excel result at all! 
Conclusion: the difference between the shallow and deep refiners doesn’t seem to be so much important for the first sight. But you have to be aware there’s a huge difference in a real production environment as you and your users might have some “hidden” refiners, and sometimes it’s hard to understand why. In other words, if a refiner value shows up on your Refinement Panel, that means: - In case of FAST Search for SharePoint (deep refiner): There’s at least one item matching this refiner value in the whole result set. Exact number of the items match the refiner value is included.
- In case of SharePoint Search (shallow refiner): There’s at least one item matching this refiner value in the first 50 results.
If you cannot see a specific value on the Refiner Panel, that means: - In case of FAST Search for SharePoint (deep refiner): There’s no result matching this refiner value at all.
- In case of SharePoint Search (shallow refiner): There’s no result matching this refiner in the first 50 results.
|
| Recently, I’ve made a webinar with Dave Coleman and MetaVis. Thanks to the great people attending, I’ve got lot more question I was able to answer then, so as an ultimate solution I promised to answer them on my blog. So here are my answers, part #1: - What is the motivation behind prevent folders under Document Sets - is it search related or just an business level decision? – Unfortunately I don’t know the exact answer. It was a technical decision during design time, as I know. The property management and all inheritance things could be much more complicated in case we could have a hierarchy inside a Document Set.
- Will you provide the ppt's after the webinar? – Yes, the PPTs have been uploaded to slideshare: http://www.slideshare.net/aghy/information-architecture-searchbettertogether-8814725
- Can the content uploaded and processed by content organizer be moved to different libraries in different site collections? – Out of the box, the Content Organizer Rules can move the documents inside a Site Collection. If you need to move them out, you need some custom solution.
- Is SharePoint 2010 search based on FAST? – No, FAST Search for SharePoint is based on the FAST Search product line, and SP2010 Search is based on the previous SharePoint products’ code.
- Can we import Term sets from external system like external data in other software running in the company like we can do this in via bcs? – Yes, you can import term sets in CSV format. You can find more information here: http://technet.microsoft.com/en-us/library/ee424396.aspx
- Have we been seeing any BA-Insight Longitude Search additions to FAST Search or was it all only FASt Search? – No, all my demos contained out-of-the-box capabilities only (both SP2010 and FAST Search).
- Is Search also limited to a single Site (Server) or could this be run over multiple Sites (Servers) at same time? – You can scale-out your Crawler and Query servers to multiple servers both in case of SP2010 Search and FAST Search.
- How do activate FAST Search? – FAST Search Server 2010 for SharePoint (FS4SP) is a separated product. Steps for “activating” FS4SP are the followings:
- You must have a SharePoint 2010 Enterprise farm.
- Install FAST Search server(s). Depending on your environment, you can have one or more FS4SP servers, logically it’s a farm too.
- Configure your SP2010 farm and the FS4SP farm to work together, create the Search Server Applications in SharePoint.
- If you’re interested in much more details, here are some useful links and troubleshooting steps for you: http://aghy.hu/AghyBlog_EN/Lists/Posts/Post.aspx?ID=119
- How to better integrate search in the custom applications (not from inside sharepoint)? – SharePoint has both an API (Object Model) and a collection of Web Services. You can call those Web Services from any remote application (with proper permissions, of course), while Object Model can be used on the SharePoint server only. You can find the QueryService reference here: http://msdn.microsoft.com/en-us/library/websvcsearch.queryservice_members.aspx
More to come soon (sonner than this one, I can tell you)… |
| Recently, I have seen several “issues” after installing SharePoint 2010 SP1. The fix is very easy, but first let me describe the symptoms I have seen. 1. “The search application ‘Search Service Application’ on server MYSERVER did not finish loading. View the event logs on the affected server for more information.” This error appears on the Search Service Application. Event Log on the server contains tons of errors like this: Log Name: Application Source: Microsoft-SharePoint Products-SharePoint Server Date: 7/19/2011 5:01:22 PM Event ID: 6481 Task Category: Shared Services Level: Error Keywords: User: MYDOMAIN\svcuser Computer: myserver.mydomain.local Description: Application Server job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (898667e4-126e-45d2-bb52-43f613669084). Reason: The device is not ready. Technical Support Details: System.IO.FileNotFoundException: The device is not ready. at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize() at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob) 2. On a different server, the SharePoint needed to be upgraded from Standard to Enterprise (after SP1 has just been installed). After entering the license key, we got an “Unsuccessful” error on the User Interface, and this in the Event Log: The synchronization operation of the search component: d289abde-9641-46d0-8d32-0345f1885704 associated to the search application: Search Service Application on server: BAIQATEST01 has failed. The component version is not compatible with the search database: Search_Service_Application_CrawlStoreDB_a737e7614f034544a8c1da6fe4a24f7b on server: BAIQATEST01. The possible cause for this failure is The database schema version is less than the minimum backwards compatibility schema version that is supported for this component. To resolve this problem upgrade this database. The reason of these errors were the same in both cases: SharePoint 2010 Config Wizard should have been run after installing SP1. Once you run the Config Wizard, these errors disappear and everything starts to work fine – again! |
| As you know I’m playing working with SharePoint/FAST Search a lot. I have a lot of tasks when I have to sit on the button F5 while crawling and check the status: is it started? is it still crawling? is it finished yet?… I have to hit F5 in every minute. I’m too lazy, so decided to write a PowerShell script that does nothing but checking the crawl status of a Content Source and writes it to the console to me. And I can work on my second screen while it’s working and working and working - without touching F5. The script is pretty easy: $SSA = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
$ContentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource -Identity "My Content Source" do {
Write-Host $ContentSource.CrawlState (Get-Date).ToString() "-" $ContentSource.SuccessCount "/" $ContentSource.WarningCount "/" $ContentSource.ErrorCount
Start-Sleep 5
} while (1) Yes, it works fine for FAST (FS4SP) Content Sources too. |
| In case of having not strong enough hardware there’s a pretty common request for start the crawl in the evening and pause in the next morning, before the work day starts. Scheduling the start of Full/Incremental Crawl is pretty easy from the admin UI, but you have to do some trick if you want to schedule the pause too. Here is my favorite trick: use PowerShell!
Here is what I do here:
- Create a script to start/resume the crawl (CrawlStart.ps1).
- Create a script to pause the crawl (CrawlPause.ps1).
- Schedule the script CrawlStart.ps1 to run in the evening (like 6pm).
- Schedule the script CrawlPause.ps1 to run in the morning (like 6am).
Is it simple, right? ;)
Here are some more details.
First, we have to know how to add the SharePoint SnapIn to PowerShell. Here is the command we need: Add-PSSnapin Microsoft.SharePoint.PowerShell.
Second, we have to get the Content Source from our Search Service Application:
$SSA = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
$ContentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource -Identity "My Content Source"
Then we have to know how to check the status of this content source’s crawl: $ContentSource.CrawlStatus. Here are the available values:
- Idle
- CrawlStarting
- CrawlingIncremental / CrawlingFull
- CrawlPausing
- Paused
- CrawlResuming
- CrawlCompleting
- CrawlStopping
Finally, we have to know how to start/pause/resume the crawling:
- Start Full Crawl: $ContentSource.StartFullCrawl()
- Start Incremental Crawl: $ContentSource.StartIncrementalCrawl()
- Pause the current crawl: $ContentSource.PauseCrawl()
- Resume the crawl: $ContentSource.ResumeCrawl()
That’s it. Here are the final scripts:
1. CrawlStart.ps1
Add-PSSnapin Microsoft.SharePoint.PowerShell
$SSA = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
$ContentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource -Identity "My Content Source"
if ($ContentSource.CrawlStatus -eq "Idle" ) {
$ContentSource.StartIncrementalCrawl()
Write-Host "Starting Incremental Crawl"
if ($ContentSource.CrawlStatus -eq "Paused" ) {
$ContentSource.ResumeCrawl()
Write-Host "Resuminging Incremental Crawl"
}
2. CrawlPause.ps1
Add-PSSnapin Microsoft.SharePoint.PowerShell
$SSA = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
$ContentSource = $SSA | Get-SPEnterpriseSearchCrawlContentSource -Identity "My Content Source" Write-Host $ContentSource.CrawlState
if (($ContentSource.CrawlStatus -eq "CrawlingIncremental" ) -or ($ContentSource.CrawlStatus -eq "CrawlingFull" )) {
$ContentSource.PauseCrawl()
Write-Host "Pausing the current Crawl"
}
Write-host $ContentSource.CrawlState
And finally, you have to schedule these tasks as a Windows job, by using these actions: powershell –command “& ‘C:\Scripts\CrawlStart.ps1’” to start and powershell –command “& ‘C:\Scripts\CrawlPause.ps1’” to pause your crawl.
Ps.: These scripts work fine for FAST Content Sources in SharePoint too, in this case you have to use the FAST Content SSA.
Enjoy! |
|
|
|
|
 |
Senior Solutions Consultant,
Project Leader
at BA Insight
SharePoint Server MVP |
Email: aghy[@]aghy[.]hu
Twitter: @molnaragnes
|
|
|
|
|
|
|
|
|
|
Hosted by
|
|
|
|
|