Filters StandardTokenizer with LowerCaseFilter and StopFilter, using a
configurable list of stop words.
| constant | type | note |
|---|---|---|
| DEFAULT_MAX_TOKEN_LENGTH | int | Default maximum allowed token length |
Builds an analyzer with the given stop words.
| name | type | description |
|---|---|---|
| stop_words | org.apache.lucene.analysis.CharArraySet | stop words |
<init>()Builds an analyzer with no stop words.
<init>(java.io.Reader stopwords)Builds an analyzer with the stop words from the given reader.
| name | type | description |
|---|---|---|
| stopwords | java.io.Reader | Reader to read stop words from |
getMaxTokenLength() · also: get_max_token_lengthReturns the current maximum token length
setMaxTokenLength(int length) · also: set_max_token_lengthSet the max allowed token length. Tokens larger than this will be chopped up at this token
length and emitted as multiple tokens. If you need to skip such large tokens, you could
increase this max length, and then use LengthFilter to remove long tokens. The default
is StandardAnalyzer#DEFAULT_MAX_TOKEN_LENGTH.
| field | type | note |
|---|---|---|
| GLOBAL_REUSE_STRATEGY | Analyzer.ReuseStrategy | A predefined ReuseStrategy that reuses the same components for every field. |
| PER_FIELD_REUSE_STRATEGY | Analyzer.ReuseStrategy | A predefined ReuseStrategy that reuses components per-field by maintaining a Map of
TokenStreamComponent per field name. |
close()Frees persistent resources used by this Analyzer
getOffsetGap(String fieldName)Just like #getPositionIncrementGap, except for Token offsets instead. By default this
returns 1. This method is only called if the field produced at least one token for indexing.
| name | type | description |
|---|---|---|
| field_name | String | the field just indexed |
Returns: offset gap, added to the next token emitted from #tokenStream(String,Reader).
This value must be >= 0.
getPositionIncrementGap(String fieldName)Invoked before indexing a IndexableField instance if terms have already been added to that field. This allows custom analyzers to place an automatic position increment gap between IndexbleField instances using the same field name. The default value position increment gap is 0. With a 0 position increment gap and the typical default token position increment of 1, all terms in a field, including across IndexableField instances, are in successive positions, allowing exact PhraseQuery matches, for instance, across IndexableField instance boundaries.
| name | type | description |
|---|---|---|
| field_name | String | IndexableField name being indexed. |
Returns: position increment gap, added to the next token emitted from #tokenStream(String,Reader). This value must be >= 0.
normalize(String fieldName, String text)Normalize a string down to the representation that it would have in the index.
This is typically used by query parsers in order to generate a query on a given term, without tokenizing or stemming, which are undesirable if the string to analyze is a partial word (eg. in case of a wildcard or fuzzy query).
This method uses #initReaderForNormalization(String, Reader) in order to apply
necessary character-level normalization and then #normalize(String, TokenStream) in
order to apply the normalizing token filters.
Returns the used ReuseStrategy.
tokenStream(String fieldName, java.io.Reader reader)Returns a TokenStream suitable for fieldName, tokenizing the contents of
reader.
This method uses #createComponents(String) to obtain an instance of TokenStreamComponents. It returns the sink of the components and stores the components
internally. Subsequent calls to this method will reuse the previously stored components after
resetting them through TokenStreamComponents#setReader(Reader).
NOTE: After calling this method, the consumer must follow the workflow described in
TokenStream to properly consume its contents. See the Analysis package documentation for some examples demonstrating this.
NOTE: If your data is available as a String, use #tokenStream(String,
String) which reuses a StringReader-like instance internally.
| name | type | description |
|---|---|---|
| field_name | String | the name of the field the created TokenStream is used for |
| reader | java.io.Reader | the reader the streams source reads from |
Returns: TokenStream for iterating the analyzed content of reader
| AlreadyClosedException | if the Analyzer is closed. | |
tokenStream(String fieldName, String text)Returns a TokenStream suitable for fieldName, tokenizing the contents of
text.
This method uses #createComponents(String) to obtain an instance of TokenStreamComponents. It returns the sink of the components and stores the components
internally. Subsequent calls to this method will reuse the previously stored components after
resetting them through TokenStreamComponents#setReader(Reader).
NOTE: After calling this method, the consumer must follow the workflow described in
TokenStream to properly consume its contents. See the Analysis package documentation for some examples demonstrating this.
| name | type | description |
|---|---|---|
| field_name | String | the name of the field the created TokenStream is used for |
| text | String | the String the streams source reads from |
Returns: TokenStream for iterating the analyzed content of reader
| AlreadyClosedException | if the Analyzer is closed. | |
Returns the analyzer's stopword set or an empty set if the analyzer has no stopwords
Returns: the analyzer's stopword set or an empty set if the analyzer has no stopwords