A TokenStream enumerates the sequence of tokens, either from Fields of a
Document or from query text.
This is an abstract class; concrete subclasses are:
Tokenizer, a TokenStream whose input is a Reader; and
TokenFilter, a TokenStream whose input is another TokenStream
.
TokenStream extends AttributeSource, which provides access to all of the
token Attributes for the TokenStream. Note that only one instance per AttributeImpl is created and reused for every token. This approach reduces object creation and
allows local caching of references to the AttributeImpls. See #incrementToken()
for further details.
The workflow of the new TokenStream API is as follows:
TokenStream/TokenFilters which add/get attributes
to/from the AttributeSource.
TokenStream#reset().
#incrementToken() until it returns false consuming the
attributes after each call.
#end() so that any end-of-stream operations can be performed.
#close() to release any resource when finished using the
TokenStream.
#incrementToken().
You can find some example code for the new API in the analysis package level Javadoc.
Sometimes it is desirable to capture a current state of a TokenStream, e.g., for
buffering purposes (see CachingTokenFilter, TeeSinkTokenFilter). For this usecase AttributeSource#captureState and AttributeSource#restoreState can be used.
The TokenStream-API in Lucene is based on the decorator pattern. Therefore all
non-abstract subclasses must be final or have at least a final implementation of #incrementToken! This is checked when Java assertions are enabled.
| constant | type | note |
|---|---|---|
| DEFAULT_TOKEN_ATTRIBUTE_FACTORY | AttributeFactory | Default AttributeFactory instance that should be used for TokenStreams. |
close()Releases resources associated with this stream.
If you override this method, always call super.close(), otherwise some internal
state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on reuse).
end()This method is called by the consumer after the last token has been consumed, after #incrementToken() returned false (using the new TokenStream API).
Streams implementing the old API should upgrade to use this feature.
This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.
Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.
If you override this method, always call super.end().
| IOException | If an I/O error occurs | |
incrementToken()Consumers (i.e., IndexWriter) use this method to advance the stream to the next token.
Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.
The producer must make no assumptions about the attributes after the method has been
returned: the caller may arbitrarily change it. If the producer needs to preserve the state for
subsequent calls, it can use #captureState to create a copy of the current attribute
state.
This method is called for every token of a document, so an efficient implementation is
crucial for good performance. To avoid calls to #addAttribute(Class) and #getAttribute(Class), references to all AttributeImpls that this stream uses should be
retrieved during instantiation.
To ensure that filters and consumers know which attributes are available, the attributes
must be added during instantiation. Filters and consumers are not required to check for
availability of attributes in #incrementToken().
Returns: false for end of stream; true otherwise
reset()This method is called by a consumer before it begins consumption using #incrementToken().
Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.
If you override this method, always call super.reset(), otherwise some internal
state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).
addAttribute(Class<T> attClass)The caller must pass in a Class<? extends Attribute> value. This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned.
addAttributeImpl(org.apache.lucene.util.AttributeImpl att)Expert: Adds a custom AttributeImpl instance with one or more Attribute interfaces.
NOTE: It is not guaranteed, that att is added to the
AttributeSource, because the provided attributes may already exist. You should always
retrieve the wanted attributes using #getAttribute after adding with this method and
cast to your class. The recommended way to use custom implementations is using an AttributeFactory.
This method will only add the Attribute interfaces directly implemented by the class and its super classes.
getAttributeClassesIterator() · also: get_attribute_classes_iteratorReturns a new iterator that iterates the attribute classes in the same order they were added in.
returns the used AttributeFactory.
getAttributeImplsIterator() · also: get_attribute_impls_iteratorReturns a new iterator that iterates all unique Attribute implementations. This iterator may
contain less entries that #getAttributeClassesIterator, if one instance implements more
than one Attribute interface.
Captures the state of all Attributes. The return value can be passed to #restoreState
to restore the state of this or another AttributeSource.
Be careful, this method comes with a cost of deep copying all attributes in the source.
clearAttributes()Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each
Attribute implementation.
Performs a clone of all AttributeImpl instances returned in a new
AttributeSource instance. This method can be used to e.g. create another TokenStream with
exactly the same attributes (using #AttributeSource(AttributeSource)). You can also use
it as a (non-performant) replacement for #captureState, if you need to look into /
modify the captured state.
Copies the contents of this AttributeSource to the given target
AttributeSource. The given instance has to provide all Attributes this instance
contains. The actual attribute implementations must be identical in both
AttributeSource instances; ideally both AttributeSource instances should use the same AttributeFactory. You can use this method as a replacement for #restoreState, if you
use #cloneAttributes instead of #captureState.
endAttributes()Resets all Attributes in this AttributeSource by calling AttributeImpl#end() on each
Attribute implementation.
equals(Object obj)getAttribute(Class<T> attClass)Returns the instance of the passed in Attribute contained in this AttributeSource
The caller must pass in a Class<? extends Attribute> value.
Returns: instance of the passed in Attribute, or null if this AttributeSource does not
contain the Attribute. It is recommended to always use #addAttribute even in
consumers of TokenStreams, because you cannot know if a specific TokenStream really uses a
specific Attribute. #addAttribute will automatically make the attribute available.
If you want to only use the attribute, if it is available (to optimize consuming), use
#hasAttribute.
hasAttribute(Class<? extends org.apache.lucene.util.Attribute> attClass)The caller must pass in a Class<? extends Attribute> value. Returns true, iff this AttributeSource contains the passed-in Attribute.
hasAttributes() · also: has_attributesReturns true, iff this AttributeSource has any attributes
hashCode()reflectAsString(boolean prependAttClass)This method returns the current attribute values as a string in the following format by calling
the #reflectWith(AttributeReflector) method:
prependAttClass=true:
"AttributeClass#key=value,AttributeClass#key=value"
prependAttClass=false: "key=value,key=value"
reflectWith(org.apache.lucene.util.AttributeReflector reflector)This method is for introspection of attributes, it should simply add the key/values this
AttributeSource holds to the given AttributeReflector.
This method iterates over all Attribute implementations and calls the corresponding AttributeImpl#reflectWith method.
removeAllAttributes()Removes all attributes and their implementations from this AttributeSource.
restoreState(org.apache.lucene.util.AttributeSource.State state)Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (e.g. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation).
Note that this method does not affect attributes of the targetStream that are not contained
in this state. In other words, if for example the targetStream contains an OffsetAttribute, but
this state doesn't, then the value of the OffsetAttribute remains unchanged. It might be
desirable to reset its value to the default, in which case the caller should first call TokenStream#clearAttributes() on the targetStream.
toString()Returns a string consisting of the class's simple name, the hex representation of the identity hash code, and the current reflection of all attributes.