An Analyzer builds TokenStreams, which analyze text. It thus represents a policy for extracting index terms from text.
In order to define what analysis is done, subclasses must define their TokenStreamComponents in #createComponents(String). The components
are then reused in each call to #tokenStream(String, Reader).
Simple example:
Analyzer analyzer = new Analyzer() {
@Override
protected TokenStreamComponents createComponents(String fieldName) {
Tokenizer source = new FooTokenizer(reader);
TokenStream filter = new FooFilter(source);
filter = new BarFilter(filter);
return new TokenStreamComponents(source, filter);
}
@Override
protected TokenStream normalize(String fieldName, TokenStream in) {
// Assuming FooFilter is about normalization and BarFilter is about
// stemming, only FooFilter should be applied
return new FooFilter(in);
}
};
For more examples, see the Analysis package documentation.
For some concrete implementations bundled with Lucene, look in the analysis modules:
| constant | type | note |
|---|---|---|
| GLOBAL_REUSE_STRATEGY | Analyzer.ReuseStrategy | A predefined ReuseStrategy that reuses the same components for every field. |
| PER_FIELD_REUSE_STRATEGY | Analyzer.ReuseStrategy | A predefined ReuseStrategy that reuses components per-field by maintaining a Map of
TokenStreamComponent per field name. |
close()Frees persistent resources used by this Analyzer
getOffsetGap(String fieldName)Just like #getPositionIncrementGap, except for Token offsets instead. By default this
returns 1. This method is only called if the field produced at least one token for indexing.
| name | type | description |
|---|---|---|
| field_name | String | the field just indexed |
Returns: offset gap, added to the next token emitted from #tokenStream(String,Reader).
This value must be >= 0.
getPositionIncrementGap(String fieldName)Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is 0. With a 0 position increment gap and the typical default token position increment of 1, all terms in a field, including across IndexableField instances, are in successive positions, allowing exact PhraseQuery matches, for instance, across IndexableField instance boundaries.
| name | type | description |
|---|---|---|
| field_name | String | IndexableField name being indexed. |
Returns: position increment gap, added to the next token emitted from #tokenStream(String,Reader). This value must be >= 0.
normalize(String fieldName, String text)Normalize a string down to the representation that it would have in the index.
This is typically used by query parsers in order to generate a query on a given term, without tokenizing or stemming, which are undesirable if the string to analyze is a partial word (eg. in case of a wildcard or fuzzy query).
This method uses #initReaderForNormalization(String, Reader) in order to apply
necessary character-level normalization and then #normalize(String, TokenStream) in
order to apply the normalizing token filters.
Returns the used ReuseStrategy.
tokenStream(String fieldName, java.io.Reader reader)Returns a TokenStream suitable for fieldName, tokenizing the contents of
reader.
This method uses #createComponents(String) to obtain an instance of TokenStreamComponents. It returns the sink of the components and stores the components
internally. Subsequent calls to this method will reuse the previously stored components after
resetting them through TokenStreamComponents#setReader(Reader).
NOTE: After calling this method, the consumer must follow the workflow described in
TokenStream to properly consume its contents. See the Analysis package documentation for some examples demonstrating this.
NOTE: If your data is available as a String, use #tokenStream(String,
String) which reuses a StringReader-like instance internally.
| name | type | description |
|---|---|---|
| field_name | String | the name of the field the created TokenStream is used for |
| reader | java.io.Reader | the reader the streams source reads from |
Returns: TokenStream for iterating the analyzed content of reader
| AlreadyClosedException | if the Analyzer is closed. | |
tokenStream(String fieldName, String text)Returns a TokenStream suitable for fieldName, tokenizing the contents of
text.
This method uses #createComponents(String) to obtain an instance of TokenStreamComponents. It returns the sink of the components and stores the components
internally. Subsequent calls to this method will reuse the previously stored components after
resetting them through TokenStreamComponents#setReader(Reader).
NOTE: After calling this method, the consumer must follow the workflow described in
TokenStream to properly consume its contents. See the Analysis package documentation for some examples demonstrating this.
| name | type | description |
|---|---|---|
| field_name | String | the name of the field the created TokenStream is used for |
| text | String | the String the streams source reads from |
Returns: TokenStream for iterating the analyzed content of reader
| AlreadyClosedException | if the Analyzer is closed. | |