FAST/ESP Views and search profiles


This article is an introduction to views and search profiles in ESP. It contains information related to problems seen with view deployment, especially on systems with many deployed views.

CAUTION: Customers should check with Technical Support before modifying any database table directly, especially on production systems.

A search view defines what content is to be searched (for example, which collections are to be searched), how the content is searched (for example, which dictionary is to be used when spell checking), and how results are returned, (for example, which navigators are to be returned).

Search profiles are settings that define what to search and how queries and results should be processed and displayed.

How are views created?
Default views are created automatically for every collection. These views are given the same name as their respective collection. The functionality of these views is to simply filter search results on the collection. Each cluster has a special view that searches all collections in the cluster. This view has the naming conversion espsystem<cluster> (by default, this view is named espsystemwebcluster).

How are Search Profiles and Views related?
A search profile always consists of two views, a "preview" view and a "published" view. In the Search Business Center (SBC), changes to a search profile are immediately deployed to the preview view, in order for users to test the changes. When the desired results are achieved, one can publish the view so that the changes are moved to the respective published view. The search application’s front-end typically only searches the published view.
The preview view of a search profile is called <search profile name>sppreview, and the published view is called <search profile name>sppublished.

The following are the system components that relate to the concept of views:
1.      Fast Home, Search Business Center and other Adminserver clients use the view service for storing, reading, modifying and launching view deployments.

2.      The Deployment Manager contains a view-deployment plug-in.  This plugin is responsible for transforming a view model that it receives from the View Service, into configuration data.  The configuration data is deployed to the DSAPI and the QRServers.  The Deployment Manager is the central component for orchestrating the configuration changes.

3.      The View Service provides functions for storing, reading, modifying and launching view deployments. The View Service is part of the Adminserver.  The View Service is responsible for generating the QT pipeline configuration for a view, together with associated resources (such as dictionaries and matcher configurations). The View Service uses the Deployment Manager to deploy these resources to the QRServers.

4.      The Dictionary Service is used by the View Service to generate automaton for dictionaries used by a view. 

5.      The Resource Service is used as a storage facility for deployed views and associated resources (for example, dictionaries).  The Resource Service does not have any built in knowledge about views.

6.      The QRServer routes queries through the view-specific QT pipeline and sets query parameters based on the view configuration. The QRServer downloads the view's and QT pipeline configurations from the Resource Service when the Deployment Manager tells it to deploy a view.

7.      The DSAPI/The search front-end reads a view/QRServer mapping resource, available from the QRServers’ HTTP interface, and optionally the view-configuration itself, which is also available from the QRServers.  It sends queries, which have been marked with the target view, to QRServers that serve that view.

View deployment
Deployment is the process of making a resource available in the Resource Service and notifying the components that need that resource about its presence. In terms of views, deployment is the process of making a view searchable on the QRServers.  

The following is a high-level overview of view deployment:

What is the relationship between views and QT pipelines?
Each view has its own QT pipeline, which by default is based on the "scopesearch" pipeline. The QRServer routes queries to the correct pipeline, based on the view query parameter. In addition to routing queries to the pipeline, the QRServer adds query parameters to the query, based on the view configuration. When a view is deployed, a copy of the scopesearch pipeline is made, and the DidYouMean and synonym query transformations for that pipeline are altered according to the settings in the view. For example, when a synonym is added to a search profile in the SBC, the preview and published views of that search profile will have a synonym QT with a synonym dictionary that includes the added synonym.

How are the properties of the view mapped to an active system configuration?
This translation happens mainly in three places:
•         In the search client - The client needs to add the view query parameter, with the name of the view as the value, to each query related to that view.

•         In the QRServer - The QRServer selects a view by reading the value of the view query parameter, and expands this into additional query parameters, based on the properties of the view.

•         In the View Service - The View Service is responsible for generating a QT pipeline based on the properties of the view, and deploying this in the form of a qtf-config.xml configuration file, to the QR-Server.

Troubleshooting View Deployment Problems.

Useful tools:
1.      view-admin (%FASTSEARCH%\bin\view-admin)
This tool can be used to create, edit, delete and deploy views from the command-line.  The view-admin tool allows users to access features of the views currently not available through the Search Business Center.
Tip: Use view-admin -h for help.

2.      qrserver-admin (%FASTSEARCH%\bin\qrserver-admin)
The qrserver-admin tool is used to control how the Adminserver and QRServers work together to deploy views.  The qrserver-admin tool allows users to control how views are deployed to the QRServers, and enables users to register and unregister QRServers.  The Adminserver maintains an internal list of all QRServers in the system.  This list is used when views are deployed to ensure that all QRServers receive the updated views.  It also prevents the system from entering an inconsistent state if one or more QRServers are down.
Tip: Use qrserver-admin -h for help.

3.      qrserverlist (http://[]:[baseport+2100]/get?qrserverlist)
The qrserverlist resource is a list of all deployed views and the QRServers where those views are available.

The initial steps one should take if a view deployment problem is encountered, include checking the status of the views as listed by the view-admin command and checking the consistency of the views in qrserverlist.

    I.      view-admin – m list

   II.      view-admin –m status

 III.       Check if the qrseverlist shows all the views:

IV. Ensure the qrserver(s) is/are registered properly and are available (%FASTSEARCH%\qrserver-admin -m status) .This command shows the status of the QRServers known to the Adminserver. This will show whether the QRServers are running, and whether their CORBA and HTTP interfaces are responding. QRServer(s) must be registered with the Adminserver for view deployment to work.

How can one sync collections and views when they are not in sync?
One can force a view to be created for each collection (if it doesn't already exist) with the following commands:
%FASTSEARCH%\view-admin -m createmissing –d

How can one view and edit the settings of a view, without using SBC?
The vespa ‘View’ table in the postgres database contains the view configuration. The entire view is stored in this table as a blob of XML. Use the view-admin tool to export the view as an XML file, perform any desired edit,  and import the view.  Import and export operations can also be used as a method to backup and restore views, and to transfer views from one system to another. For example:
view-admin -m export -V myview
view-admin -m import -V myview.xml -o -d

How can one determine the count of deployments, and their status?
Log into the vespa db
1. Execute the command: cd %FASTSEARCH%\rdbms\bin
2. Execute the command: psql --username fast -p 16070 vespa (If prompted for password enter ‘fast’)
3. Execute the command: select status,count(status) from DM_Deployment group by status;

The above statement will give the output of counts and status codes. Use the below code keys to determine the status:
STATUS_READY = 1 (deployments that are ready to be deployed.)
STATUS_WAITING = 2 (deployments that are waiting to be deployed)
STATUS_RUNNING = 3(deployment operations that are running.)
STATUS_FINISHED_OK = 4 (deployment operations that have finished successfully)
STATUS_FINISHED_FAILED = 5 (deployment operations that have failed)

What actions should one take if the number of views (count) is high, and views cannot be deleted or deployed?
Generally, there is no defined maximum number of views which a cluster/system may have. This is because the number of search profiles that can be supported by one system is dependent on multiple factors, such as the dictionaries, navigators, linguistics, rank profile, result views, etc., for each profile.  

IMPORTANT: We do not recommend deploying a high number of views (greater than 100) on a cluster/system. From experience, a high number of views may result in problems during:
-deploying views
-deleting collections
-updating index-profile
-deleting views
-deploying dictionaries

The general best practice is to maintain approximately 100 views or fewer (50 Search Profiles) per cluster.
A workaround in this situation is to delete all deployments tables, delete any views not needed, and then deploy the needed views, using the view-admin tool using the below steps: 

Log into vespa DB from command line:

1. Execute the command: cd %FASTSEARCH%\rdbms\bin
2. Execute the command: psql --username fast -p 16070 vespa
3. After logging into vespa, delete the following deployment tables by running below commands:
delete from DM_ResourceProperty;
delete from DM_Resource;
delete from DM_DeploymentProperty;
delete from DM_DeployedObject;
delete from DM_Deployment; 

After the above steps, view-admin status should show all views undeployed:

NOTE: It is recommended for this process to be performed during a maintenance window because truncating the deployment tables will bring search down and trying to fetch the views will result in the error below:

At this point, unwanted views can be deleted. One can then deploy only the needed views. If there are many views, one can deploy them individually to avoid any timeouts during deployment. i.e: %FASTSEARCH%\bin\ view-admin -m deploy -w <view1> <view2> <view3> ......

In addition, it is a common practice to only deploy ‘published’ views, as they are the ones used by search front ends. This will help reduce the total number of deployed views.


Artikelnummer: 2587443 – Letzte Überarbeitung: 20.11.2012 – Revision: 1