Maximum token count
By default, full-text indexing is applied to complete documents. To improve indexing performance, you can use the maximum token count.
If all relevant search terms are located at the beginning of a document, for example in the table of contents or in the foreword, it may be useful to limit full-text indexing. In the example below, a filter in the field type definition is used to consider only the first 1000 different tokens (maxTokenCount). This setting can be used for any field type.
<fieldType name="text_stemming_de" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LimitTokenCountFilterFactory" maxTokenCount="1000"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_de.txt"/>
<filter class="solr.WordDelimiterGraphFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1" types="lang/wdfftypes.txt"/>
<filter class="solr.LowerCaseFilterFactory"/>
<!-- Optionally you can use another stem filter with different aggressive aproach of stemming
<filter class="solr.GermanStemFilterFactory"/>
<filter class="solr.GermanLightStemFilterFactory"/>
Setting the maximum token count is useful only if many large documents containing many tokens enter the indexing process. It does not make sense for many small documents.
Complete indexing of large documents consumes memory and processing time and slows down the overall indexing process.
If you expect very large documents in your environment, your specific performance requirements must be reflected in an adequate hardware infrastructure or in an appropriate search architecture, for example a distributed search environment with several shards [Wiki DSearch].
The combination of basic authentication with distributed search is impossible because of a known Solr defect (see https://issues.apache.org/jira/browse/SOLR-15237).
Fine tuning for search modes
The following full-text-related search modes can be distinguished. They are handled by different internal fields. These fields are indexed and can be further customized by Solr mechanisms:
-
Fields used by standard search queries
This search is handled by the internal fields “fulltext” and “unitedmetadata”. Standard full-text search supports language-dependent word stemming (see also the language setting in solrcore.properties in chapter Basic configuration – solrcore.properties) and removes common stop words. Solr provides different word stemmers, and the default stop word list can be customized. In this case, additional internal fields are used, for example “fulltext_en” and “unitedmetadata_en” for documents that have been detected as English by the standard search query.
-
Phrase search
Phrase search is handled by the internal fields “fulltext_phrase” and “unitedmetadata_phrase”. A phrase search is triggered by using double quotation marks around a phrase in the full-text search field.
-
Wildcard search
The wildcard fields are required to support wildcard search. The asterisk and question mark wildcards are supported in the full-text search field. In the Solr configuration of the assembly release, no special fields are defined. In a standard installation, the phrase search fields are used. It is possible to activate special wildcard fields that are optimized for queries with leading wildcards. In this case, enable the stored and indexed properties of the fields “fulltext_wildcard” and “unitedmetadata_wildcard” in the schema.xml configuration file.
To reduce memory and CPU usage, you can selectively activate or deactivate indexing for the different search modes. In the fields section of the schema.xml configuration file, set the field attributes indexed and stored to false.
The following fields must remain configured with stored="true":
-
id (Solr uniqueKey)
-
documentid
-
revisionid
These fields are required so that ImageMaster can correctly interpret and process the Solr search result. If a field is configured with indexed="true" but not with stored="true", the document cannot be mapped to the corresponding revision at all, meaning the document will not be found and therefore will not appear in the result list.
This is the minimum requirement for search result processing. Depending on the use case, additional fields may also need stored="true". For example, highlighting and snippets require the corresponding content or full-text fields to be stored.
...
<fields>
...
<field name="fulltext_phrase"
type="text" multiValued="true" indexed="false" stored="false" />
<field name="unitedmetadata_phrase"
type="text" multiValued="true" indexed="false" stored="false" />
<field name="fulltext_wildcard"
type="text_wildcard" multiValued="true" indexed="false" stored="false" />
<field name="unitedmetadata_wildcard"
type="text_wildcard" multiValued="true" indexed="false" stored="false" />
...
</fields>
...
By deactivating an index, the corresponding search mode no longer supports full-text search.
It is possible to design customization scenarios in which one of the indexes is shared for several purposes, for example by using either the phrase search index or the standard full-text index for both search modes. However, such a customization must always be developed according to project-specific requirements and should only be implemented in cooperation with your contact at T-Systems (see also section Internal configuration management and customization).