org.apache.lucene.analysis.Tokenizer

close

NOTE: The default implementation closes the input Reader, so be sure to call super.close() when overriding this method.

reader=

Expert: Set a new reader on the Tokenizer. Typically, an analyzer (in its tokenStream method) will use this to re-use a previously created tokenizer.

reset

add_attribute

The caller must pass in a Class<? extends Attribute> value. This method first checks if an instance of that class is already in this AttributeSource and returns it. Otherwise a new instance is created, added to this AttributeSource and returned.

add_attribute_impl

Expert: Adds a custom AttributeImpl instance with one or more Attribute interfaces.

NOTE: It is not guaranteed, that att is added to the AttributeSource, because the provided attributes may already exist. You should always retrieve the wanted attributes using #getAttribute after adding with this method and cast to your class. The recommended way to use custom implementations is using an AttributeFactory.

This method will only add the Attribute interfaces directly implemented by the class and its super classes.

attribute_classes_iterator

Returns a new iterator that iterates the attribute classes in the same order they were added in.

attribute_factory

returns the used AttributeFactory.

attribute_impls_iterator

Returns a new iterator that iterates all unique Attribute implementations. This iterator may contain less entries that #getAttributeClassesIterator, if one instance implements more than one Attribute interface.

capture_state

Captures the state of all Attributes. The return value can be passed to #restoreState to restore the state of this or another AttributeSource.

Be careful, this method comes with a cost of deep copying all attributes in the source.

clear_attributes

Resets all Attributes in this AttributeSource by calling AttributeImpl#clear() on each Attribute implementation.

clone_attributes

Performs a clone of all AttributeImpl instances returned in a new AttributeSource instance. This method can be used to e.g. create another TokenStream with exactly the same attributes (using #AttributeSource(AttributeSource)). You can also use it as a (non-performant) replacement for #captureState, if you need to look into / modify the captured state.

copy_to

Copies the contents of this AttributeSource to the given target AttributeSource. The given instance has to provide all Attributes this instance contains. The actual attribute implementations must be identical in both AttributeSource instances; ideally both AttributeSource instances should use the same AttributeFactory. You can use this method as a replacement for #restoreState, if you use #cloneAttributes instead of #captureState.

end_attributes

Resets all Attributes in this AttributeSource by calling AttributeImpl#end() on each Attribute implementation.

equals

get_attribute

Returns the instance of the passed in Attribute contained in this AttributeSource

The caller must pass in a Class<? extends Attribute> value.

Returns: instance of the passed in Attribute, or null if this AttributeSource does not contain the Attribute. It is recommended to always use #addAttribute even in consumers of TokenStreams, because you cannot know if a specific TokenStream really uses a specific Attribute. #addAttribute will automatically make the attribute available. If you want to only use the attribute, if it is available (to optimize consuming), use #hasAttribute.

has_attribute

The caller must pass in a Class<? extends Attribute> value. Returns true, iff this AttributeSource contains the passed-in Attribute.

has_attributes?

Returns true, iff this AttributeSource has any attributes

hash_code

reflect_as_string

This method returns the current attribute values as a string in the following format by calling the #reflectWith(AttributeReflector) method:

• iff prependAttClass=true: "AttributeClass#key=value,AttributeClass#key=value"
• iff prependAttClass=false: "key=value,key=value"

reflect_with

This method is for introspection of attributes, it should simply add the key/values this AttributeSource holds to the given AttributeReflector.

This method iterates over all Attribute implementations and calls the corresponding AttributeImpl#reflectWith method.

remove_all_attributes

Removes all attributes and their implementations from this AttributeSource.

restore_state

Restores this state by copying the values of all attribute implementations that this state contains into the attributes implementations of the targetStream. The targetStream must contain a corresponding instance for each argument contained in this state (e.g. it is not possible to restore the state of an AttributeSource containing a TermAttribute into a AttributeSource using a Token instance as implementation).

Note that this method does not affect attributes of the targetStream that are not contained in this state. In other words, if for example the targetStream contains an OffsetAttribute, but this state doesn't, then the value of the OffsetAttribute remains unchanged. It might be desirable to reset its value to the default, in which case the caller should first call TokenStream#clearAttributes() on the targetStream.

to_string

Returns a string consisting of the class's simple name, the hex representation of the identity hash code, and the current reflection of all attributes.

end

This method is called by the consumer after the last token has been consumed, after #incrementToken() returned false (using the new TokenStream API). Streams implementing the old API should upgrade to use this feature.

This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but a WhitespaceTokenizer was used.

Additionally any skipped positions (such as those removed by a stopfilter) can be applied to the position increment, or any adjustment of other attributes where the end-of-stream value may be important.

If you override this method, always call super.end().

Throws

increment_token

Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use #captureState to create a copy of the current attribute state.

This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to #addAttribute(Class) and #getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.

To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in #incrementToken().

Returns: false for end of stream; true otherwise