About Me

A career professional with 19 years of experiences with in application development, solution architecture, management and strategy. Architected, planned, and executed, many programs, projects and solutions delivering valued results to business clients and customers. Able to provide creative solutions to solve problems and articulate the value proposition to upper management and technical design to IT staff. Collaborates across teams and other facet of IT (i.e. operations, infrastructure, security) to validate solutions for completeness. Interface with clients to gather feedback, advise, and solution in the business language.

Wednesday, December 1, 2010

Using Search as an Application

Using Search as an Application


Kenneth Cooper

Enterprise SharePoint Architect


Viji Anbumani

SAI Innovations Inc

Business Case

A department came to IT with a request of a document management system that would allow them to meta-tag document (fig 1.1) and discover the content through advanced search. Our users had four document types that they wanted to start with. Each document type had more then twelve pieces of associated metadata. The metadata within the document type covered just about all the basic column types (single line, check box, single select, multi-select, Numbers, and dates). The security for the document types varied between them.

After hearing the initial requirements we knew that SharePoint was a natural fit. So the first thing we did with the users was a discovery phase to hear more of what our users wanted. This phase was also interactive; we informed our users on what SharePoint can do. The first thing that we discovered when showing our user SharePoint was they didn’t like the out of the box advanced search. They really wanted an advanced criteria screen that was reflective of their metadata in look and functionality. They wanted fixed fields that they can populated. The fields also had to have the same functionality as the metadata entry screen. So if they had a multi-select field in the metadata they wanted the same thing within the search criteria screen. The search results they wanted them to come back in a grid. They wanted to have two views for the grid compact and detailed.

The timeline and budget for this project were both short. They wanted to be up and running no later then the first of January. We started the discovery phase mid-September. But they wanted to start uploading content by the first of November. Even though the majority of the project is out of the box we also had to account for rework time.

The Challenge

The first challenge we needed to resolve was the metadata search on the different document types. After that we needed to do something about the advanced search. The easy solution was to have a custom webpart written to handle the search. Given all the other criteria of their requirements we didn’t have time to push SharePoint aside and do a custom web application. Ninety percent of the functionality was there in SharePoint already that’s why we decided to just do custom webparts.

The Plan

The department handed us excellent requirements for the document types. From pass experience with others projects as the project draws closer and the users start to see the product the rework sometime increases. This is an important factor in SharePoint because there might certain things that the users don’t like and it takes to much effort to rewrite or rework it. First a project plan was created and it had two tracks; structure and portal functionality. The structures were the work that we were doing for the site definitions, content types, and deployment packages.

The structure had to precede the portal work any way so this track was devised to get the users uploading content by November. The custom work like branding and the search web parts were part of the second track. This plan let the users know what they were getting and when.

Side note

Don’t assume that your users won’t change their mind about there fields or field types. To help keep our production SharePoint environment clean we go through (development to QA to production). For moving items between these environments there are various methods and tools. We choose to do a site definition and content type features for our document libraries. When using feature we can easily move from environment to environment. We also used import and export site command from stsadmn. The only issue with this is that if you have an error or bad data you have to take care of it once it is deployed. Cleaning up a site in production can prove to be very messy and can cause issues.

The Solution

The Search Components we used

Content Source –A source of content that you want SharePoint to crawl and index.
Managed Properties – Metadata fields that are defined in SharePoint that are indexed and put into a separate database. These fields can be used in actual search queries.
Scopes – Creates a filtered view of the content in an index based on rules defined within the SSP.

Custom Search Webpart – A custom search criteria screen was built to render the search criteria screen for the users. Another webpart was built to display the results in grid format.

Document Types & Content Types

For each document type we created a content type and a document library (see fig 1.0). The content types were applied to the appropriate document libraries. We leveraged the power of the content type field as a managed property to create our view of the department’s content. A managed property called ‘MPContentType’ was created using content type field. Next a scope was created and rules were added that specified the managed property ‘MPContentType’ as the filter. A rule was added for each content type that we wanted to add to the scope. With the scope in place, queries can be issues; resulting in a query that only queries the specified content types (i.e. Software) within the scope. Now we can isolate the content that we want to search on giving us the base line to our solution (see fig 1 ‘The Big Picture’).

More Managed Properties

The next piece of the solution was the additional managed properties that were created. We created a managed property for each piece of metadata that was associated to one of the four content types. This allowed us to do our pinpoint accurate searches on the content. The last piece of the puzzle was the advanced search webpart and the results page. Our users told us that they were only starting with four document types, but a couple of months down the road they want to add more document types. This means the design for the search criteria screen had to grow with user demand.

So that the webpart would be extensible in the future as per one of IT’s requirement was it must work with XML. The webpart takes XML as a parameter and dynamically generates the search criteria screen. That makes this a completely generic solution that can be applied over and over again without having to going back to development. For the webpart we commissioned SAI Innovations a consultant company with a strong background in SharePoint development and solutions.

Through the XML various features can be controlled and the user interface is reflective of the data entry screens. Fields with drop downs and SharePoint lists as data sources can be presented in the same way on the search criteria screen. This gives the end users a lot of flexibility and easy of use so they want have to understand and/or and other operations. Also there’s a drop down for the document type. As the document type is change the fields on the search screen are changed to match the search criteria fields for that document type.

Earlier I mentioned that the document types vary in security requirements. Since SharePoint search engine is security trimmed this did not present a problem to us. We just created SharePoint groups and modified the document libraries security to these groups and configurations.

The search results screen returned the results within a grid per request of our users. For the grid they wanted two modes a summary and a detailed view. On the detailed view it would expand the grid and show the all metadata fields for that type. The results pane also incorporated sorting and paging.

Some issues that we ran into are that the document has to be checked in for that document to be indexed.

Fig 1.0

Fig 1.1

Fig 1.2

Advanced Search Result Set

Fig 1.3

Search Quick Tutorial

The Search for the Search Configuration

When a SharePoint (MOSS) farm is setup a default Shared Service Provider (SSP for short) is created within Central Administration (might vary with WSS). It is within the SSP where the search configurations can be modified. If this is still foreign try to find a good SharePoint Admin.

As items are put into SharePoint there’s a process out there the opens certain files and reads the content and it metadata and writes out what it saw to disk and database (Ok I did skip a few steps but another paper another time). This is called the indexing process (see fig 2). The search functionality can be configured within the Shared Service Provider (SSP).

How does SharePoint know what to indexing? Within the search configuration you’ll see the content sources. A content source is content that you SharePoint to index so that it can be searchable from SharePoint. These content sources are the destination. Content sources can be created for other web sites, files shares, lotus notes, public folders, and SharePoint sites (see fig 3). Once a content source is created you can tell SharePoint to go crawl it. Also crawl schedules can be setup so new content gets indexed periodically. Since our users wanted new content to show up very quickly we created a content source for them and created an incremental schedule to index more frequently. A default content source is already there. Once we add our site to the new content source the old needs to be removed from the default one.

The indexing process also crawl’s the document’s metadata within SharePoint. This is huge piece of the solution puzzle. These metadata fields are store in an actual database. Once they have been indexed this information can be used in queries (SQL like queries). How do I expose this field to SharePoint; by making it a managed property. There would still be some work at the site level but after a few configuration changes you can make these fields show up in advanced search (the out of the box one).

Another awesome thing that you can do within the search configuration is create scopes. Think of scopes as a filtered view of the data that is in the index file. You already see this in most sites already (fig 4). If you have a drop down list in front of the search text box you’ve get scopes! One scope is ‘All Sites’ and usually the other is ‘This Site’. The All sites search all of the SharePoint content but the ‘This Site’ scope only search pertaining to that site. This feature helps scope down our target area that we want to search.

Indexing Engine
Fig 2

Fig 3

Index and Scope

Fig 4

Monday, August 9, 2010

Application Pools pertaining to SharePoint

Application Pool

Application pools are a security and process boundary for a set of IIS web applications. Multiple web application can run in a single worker process. Application pools should be create if processes need to be isolated for stability or security. Especially if the web application is going to be going to be used with custom code or has request and proceses intensive sites.

When should an application pool be created?
An application pool should be created if an application has custom code or has multiple memory and processes intensive sites. Also if the application is deemed missing critical and uptime is the utmost importance then this might warrant an app pool.

More background Information:

The Http.sys is responsible for getting http request and send responses back to users.

Http.sys -----> Listen --> Queues -->(Isolation mode send to worker process) --> Responses

Worker Process -> A processes that runs in user mode to process a request from the Http.sys queue. The worker process runs in it's own process space so it want interfere with other http processes that are on a different process. After the worker process processes the request it use Http.sys to send it response back.

Tuesday, June 29, 2010

Quick Tip Consolidating logs

If you have multiple web front ends and application servers then you know that it's painful looking through the logs. Just a quick trick that I can up with is to move the logs into one spot.
I know that you can create a shared drive and map it to all the wfe and apps but this can slow down performance so I just created a little cmdlet and run it from the App server.

My logs directory run the following

Another good practice is to move your logs from the C drive to a different drive (physical if possible). The C drive already has a lot of resource competing this will free it up and make sure that everything is still working. My general experience is when the C drive is full basically you're app is a ticking time bomb and it's going to blow up soon.

To change the logs go to Central Admin > find log and change the location to a separate (physical drive is the best) drive.

Content Type Hub Sync & Lookup lists

After the first blog post of Content Type & the Managed Metadata Service I went and checked my content type in the subscription site and discovered a fields missing. It was missing the lookup field.

To help find the cause I went to (Root Site) > Site Actions > Site Settings

Click on the content type publishing error log and you'll see a log of issuse .

To resolve the issuse I tried to export the lookup list by saving it as a list template and moved it to the subscription site. I had some trouble with this so I just recreated the list with the same name and data and IT WORKED. I was surprise because I thought it would not work because usually lookup list field use the GUID to the lookup list. The field and the data came it fine.
To keep the lists in sync there a lot of third party solutions.

Monday, June 21, 2010

SharePoint 2010 Content Type Hub & the Managed Metadata Service

SharePoint 2010 Content Type Hub & the Managed Metadata Service

What is it about:

With SharePoint 2007 I can across users that wanted to deploy content type across multiple site collections. The reasons for this is numerous (governance, size of content database, etc).

Fictitious Scenario:

MediaXYZ just installed SharePoint 2010, they have four departments but they all work print materials. As the print material (e-magazines, brochures, and papers) goes through a life-cycle starting in Global Strategy -> Design -> Marketing -> Sales.


Each department works independently of each other but management want everyone to have the same metadata and structures for the document types to make searching easier. In MOSS 2007 a content type would be used to accomplish this but it would not go beyond the site collection unless you used a feature to deploy it to multiple site collections.


Web Application and Web Site Collections

Global Strategy








Create the following content type only in http://office10/sites/GlobalStrategy

Content Type - Print Material

Print Type

Choice [Magazine, Paper, Brochure]


Single line

Client Type

Choice [Pharmaceutical, Real-Estate, Software]

Main Topic

Single line

Publish Date



Single line


In SharePoint 2010 has a new service that saves us from this insanity of features or custom code. The new service is called the Managed Metadata service application. Basically a site is designated
as the content type hub and content types from the site collection are published throughout the other site collections. The best thing about this solution is if the content type changes then all that needs to be done is to republish the content type in the hub site.


Since Global Strategy is the primary spot for the content type it will be designated the content type hub.

SharePoint Administrator
to the front line

Central Administration > Services Applications

Make sure the Managed Metadata service is associated with a web application (if needed I will post this).

DON'T click on the link for the Managed Metadata Service, click beside the name.

Next click on the properties button at the top

Set the following information to your farm specification.

In the Content Type hub section put the URL (http://office10/sites/GlobalStrategy) of the site collection that the content type will be published from. Also check off Report syndication import errors.

Click OK and return back to the Service Applications now click beside the Metadata Service Connection (Remember DON'T click on the hyperlink) next click the properties button.

Now the connection will be set click checkbox next to "Consumes content types from the Content Type
Gallery at http://office10/sites/GlobalStrategy

Go to the http://office10/sites/GlobalStrategy/_layouts/mngctype.aspx and there will be a new option will be available when you go to the content type. The new link "Manage publishing for this content type" click and publish the content type.

(The first time the Publish option will be available select that one)


The content type will NOT SHOWUP right away because content publisher and the content subscription are based on timer jobs.

Time for Timer ....

Kicking off the timer jobs go to
Monitoring > Review Job Definitions

Look for the following timer jobs, click on each link and click the "run now" button.

Now go to each site collection and there should be the content type Print Material.


Wednesday, January 27, 2010

Enabling and disabling property promotion

Recently while doing a project we have to migration document from a disk to SharePoint. We didn't have budget to buy a tool for the migration so we had to go it alone. We wrote our own tool but during the migration we ran into one problem. The metadata properties on the office documents where coming in wrong.

XYZ.doc (We wanted the title = 'Test Document') when this document was migrated it would show up like XYZ.doc title = 'ABC Document' .

After a little digging we discovered it was the property promotion of SharePoint.

I came across some powershell script and here's my version ...

param($url=$(Throw "Parameter missing: -url"), $switch=$(Throw "Parameter missing (on/off): -switch"))
"URL -> $url, swith-> $switch"
$site = New-Object Microsoft.SharePoint.SPSite($url)

$enabled = $true
if ($switch -match "off") {$enabled=$false};
"Current Setting;"+$site.RootWeb.ParserEnabled;
$site.RootWeb.ParserEnabled = $enabled
"After Setting;"+$site.RootWeb.ParserEnabled;

"Current Setting;"+$site.ParserEnabled;
#$site.ParserEnabled = $enabled
#"After Setting;"+$site.ParserEnabled;

foreach ($web in $site.AllWebs)
"Current Setting;"+$web.ParserEnabled;
$web.ParserEnabled = $enabled;
$web.Url +"["+$web.ParserEnabled+"]";
"After Setting;"+$web.ParserEnabled;