Using Advanced Database Crawler

Hello Friends, recently i was working on of the most popular shared source module for sitecore called Advanced Database Crawler developed byAlex Shyba and I must say it is awesome.

ADC uses Lucene Index as a base. Lucene is an open source search engine used in Sitecore CMS for indexing and searching the contents of a Web site. Sitecore implements a wrapper for the Lucene engine which has its own API.

The original API (Lucene.Net) and the Sitecore API (Sitecore.Search) are both accessible to developers who want to extend their indexing and search capabilities.


Note:
The Sitecore.Data.Indexing API was deprecated in Sitecore CMS 6.5 and in 7.0 it will be completely removed. So Advanced Database Crawler is using new Sitecore.Search API only.

If you have not used it then here are few quick steps to setup the module and quickly make the search work.

First you need to download the source code and then compile the project that should give you Sitecore.SharedSource.Search.dll, add reference of the DLL to you sitecore project.


Next step is to setup lucene index file configuration.

Advanced Database Crawler Configuration

Below is a brief description about each node within the configuration.

Index Element Description
<configuration>
Contains index definitions and their configuration settings.
<indexes>
The section under configuration where you define indexes. For example, ‘system’ is one index defined here.
<analyzer>
Specify what type of analyzer to use by default in the indexes. An index can use a custom analyzer or refer to this default analyzer definition.Search indexes use the same analyzer both for indexed data (documents) and for the search queries.
<categorizer>
Specify what type of analyzer to use by default in the indexes for categories in search results.
<categories>
This section is used to categories search results. It’s used for content tree search introduced in Sitecore 6.The search box is located right above the content tree in content editor.
<locations>
Specify which database you want to add to the index.
e.g.

<locations hint="list:AddCrawler">

It’s possible to have multiple locations for one index. Moreover it’s even possible to have content from different databases in the same index. Every child of the locations node has its own configuration for a particular part of the content. A name of location node is not predefined. You’re welcome to name it the way you want.You can also define the following settings under locations:

  • Database
  • Root
  • Tags
  • boost
<database>
Specify which database you want to index, for example ‘master’.
<root>
Specify the root node of the content tree to be included into the index. The indexing crawler will index content below this location
<tags>
You can attach a string tag to items from this location making it possible to filter or categorize results during a search.
<boost>
Use boost to adjust the priority of results relative to results from other locations.
<IndexAllFields>
Tells the crawler to put a copy of all the fields in the item into the index. This makes fine-grained filtering possible but creates a performance overhead.Default setting = true
<include hint="list:IncludeTemplate">
    <template>
        {GUID}
    </template>
</include>
Explicitly Include the template for indexing; Provide GUID for the template and unique tag name, it should be unique template name.
<include hint="list:ExcludeTemplate">
    <template>
        {GUID}
    </template>
</include>
Explicitly Exclude the template for indexing; Provide GUID for the template and unique tag name, it should be unique template name.
<include hint="list:IncludeField">
    <fieldId>{8CDC337E-A112-42FB-BBB4-4143751E123F}</fieldId>
</include>
If

<IndexAllFields> 

is not added then explicitly includes the field for indexing; Provide GUID for the field name and unique tag name, it should be unique field name.

<include hint="list:ExcludeField">
    <fieldId>{8CDC337E-A112-42FB-BBB4-4143751E123F}</fieldId>
</include>
If

<IndexAllFields>

is not added then explicitly excludes the field for indexing; Provide GUID for the field name and unique tag name, it should be unique field name.

<fieldCrawlers hint="raw:AddFieldCrawlers">
     ……………
</fieldCrawlers>
Custom Field crawlers for the complex field types like DropLink, DateTime, Date, and Number. If you create your custom field type then you can create your custom field crawler for that field like below.

<fieldCrawler type="Sitecore.SharedSource.Search.FieldCrawlers.LookupFieldCrawler,Sitecore.SharedSource.Search" fieldType="Droplink" />
<fieldTypes hint="raw:AddFieldTypes">
    ……………
</fieldTypes>
Here all the available field types in Sitecore are listed that needs to be indexed. If a field type is not defined, defaults of storageType=”NO”, indexType=”UN_TOKENIZED” vectorType=”NO” boost=”1f” are applied.

<fieldTypes hint="raw:AddFieldTypes">
<!-- Text fields need to be tokenized -->
    <fieldType name="single-line text" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="multi-line text" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="word document" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="html" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="rich text" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="memo" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="text" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
<!-- Multilist based fields need to be tokenized to support search of multiple values -->
    <fieldType name="multilist" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="treelist" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="treelistex" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
    <fieldType name="checklist" storageType="NO" indexType="TOKENIZED" vectorType="NO" boost="1f" />
<!-- Legacy tree list field from ver. 5.3 -->
    <fieldType name="tree list" storageType="NO"   indexType="TOKENIZED" vectorType="NO" boost="1f" />
</fieldTypes>
<dynamicFields hint="raw:AddDynamicFields">
    ……………
</dynamicFields>
If you want add custom fields to the index or dynamically add data to the existing field then you can use dynamic fields as shown on the left.Here type=”Sitecore.SharedSource.Search.DynamicFields.MyField,Sitecore.SharedSource.Search” defines the class to use for indexing the specific field. Inherit the class from BaseDynamicField and override the method ResolveValue which should always return string values.
<Monitor Changes>
Tells the crawler to subscribe to item changes and updates the index automatically. Default setting = true

Few quick information that will help you under stand features of Advanced Database crawler …

  • Multiple Indexes can be added between
    <indexes hint="list:AddIndex">…………… </indexes>

    section.

  • Individual index configuration starts with tag
    <index id="system" type="Sitecore.Search.Index, Sitecore.Kernel">
  • The id attribute is used to distinguish different indexes.
  • Parameter with desc=”folder” defines the folder name for the index.
  • Following setting defines locations for the index:
    <locations hint="list:AddCrawler">

    – Location is nothing but logical grouping of indexed data. You can use this locationid when using ADC api while searching index files.

    • It’s possible to have multiple locations for one index. Moreover it’s even possible to have content from different databases in the same index. E.g. you can have data from news and blog articles under same location or you can have different location for news and blog article.
    • Every child of the locations node has its own configuration for a particular part of the content.
    • A name of location node is not predefined. You’re welcome to name it the way you want. For example:
      <locations hint="list:AddCrawler">
             <yourownname type="Sitecore.Search.Crawlers.DatabaseCrawler, Sitecore.Kernel">
                 ………    ………
             </yourownname>
      </locations>
      
  • Every location section has
    <Database>

    section. It defines indexing database for the location.

    <Database>master</Database>
  • Every location section has
    <Root>

    section. It defines indexing database for the location.

    <Root>/sitecore/content/home</Root>
  • Every location section has
    <include>

    section. Here it’s possible to add templates items of which should be included to the index or excluded from it.

    <include hint="list:IncludeTemplate">
        <sampletemplate>{BDB6FA46-2F76-4BDE-8138-52B56C2FC47E}</sampletemplate>
    </include>
    
    <include hint="list:EncludeTemplate">
        <samplelayout>{BDB6FA46-2F76-4BDE-8138-52B56C2FC47E}</samplelayout>
    </include>
    

    Note:
    1) It does not make sense to use both of the above settings for the one location. Use only one of them.
    2) Use different tag name for the Include and Exclude template else it will only pick the last one.

  • Every location section has
    <fieldCrawlers hint="raw:AddFieldCrawlers">

    section. Here we can write our custom logic to index the special/complex fields.

    <fieldCrawlers hint="raw:AddFieldCrawlers">
           <fieldCrawler type="Sitecore.SharedSource.Search.FieldCrawlers.LookupFieldCrawler,Sitecore.SharedSource.Search" fieldType="Droplink" />
    </fieldCrawlers>
    

    Note: You can read more about Custom Field indexer in this post.

  • Every location section has
    <dynamicFields hint="raw:AddDynamicFields">

    section. Using this section you can create custom field in the Lucene index which are not created or crawled by Lucene index by default and you can put custom values within these fields.
    Note :
    You can create dynamic field and in the name=”_content” attribute you can provide name of existing field that enable to append content to the existing field values.

  • Every location section has
    <tags>

    section. Here you can tag indexed content and use it during the search procedure.

    <Tags>taskoption</Tags>
  • Last available tag under Location section is
    <Boost>

    , here you have an ability to boost indexed content among other content that belongs to other locations.

    <Boost>1.9</Boost>

Trouble shooting

While setting up the index you may encounter few issues that i faced.
1. Remove unwanted results from search – Use <includetemplate> and <excludetemplate> to get results for items from specific templates.

2. While i was setting up index for my production server somehow my index was not getting built and i tried every bit of configuration that i could think of. But finally i found the missing piece and it was enabling History engine on web database. By default history engine is not enabled on web database. You can enable it as follows.

<database id="web" 
    <Engines.HistoryEngine.Storage>
       <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.Kernel">
          <param connectionStringName="$(id)"/>
          <EntryLifeTime>30.00:00:00</EntryLifeTime>
       </obj>
    </Engines.HistoryEngine.Storage>
    <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
    …
</database>

3. Make index work on Webfarm – When you have more then one content delivery servers in that case you will have separate index files each server. In that case set following property to true.

<setting name="Indexing.ServerSpecificProperties" value="true"/>
Still having trouble making indexing work on let us know or else You can visit detailed trouble shooting post by Alex here

Looking forward for feedback

Friends this being my first blog article kindly posts your comments/suggestions/questions.
Many more articles to follow.


Also i would like to thank Kiran Patil , Parag Daraji and Brijesh Patel who provided me valuable feedback. Thank you guys 🙂

Keep visiting.
Advertisements
This entry was posted in Using Shared Source modules and tagged , , . Bookmark the permalink.