Bayesian smoothing using Dirichlet priors as implemented in the Indri Search engine (http://www.lemurproject.org/indri.php). Indri Dirichelet Smoothing!
tf_E + mu*P(t|D) P(t|E)= documentLength + documentMu mu*P(t|C) + tf_D where P(t|D)= doclen + mu
A larger value for mu, produces more smoothing. Smoothing is most important for short documents where the probabilities are more granular.
<init>(org.apache.lucene.search.similarities.LMSimilarity.CollectionModel collectionModel, boolean discountOverlaps, float mu)Instantiates the similarity with the provided parameters.
<init>(org.apache.lucene.search.similarities.LMSimilarity.CollectionModel collectionModel, float mu)Instantiates the similarity with the provided μ parameter.
<init>(float mu)Instantiates the similarity with the provided μ parameter.
<init>(org.apache.lucene.search.similarities.LMSimilarity.CollectionModel collectionModel)Instantiates the similarity with the default μ value of 2000.
<init>()Instantiates the similarity with the default μ value of 2000.
getMu() · also: get_muReturns the μ parameter.
getName() · also: get_namecomputeNorm(org.apache.lucene.index.FieldInvertState state)Computes the normalization value for a field at index-time.
The default implementation uses SmallFloat#intToByte4 to encode the number of terms
as a single byte.
WARNING: The default implementation is used by Lucene's supplied Similarity classes, which means you can change the Similarity at runtime without reindexing. If you override this method, you'll need to re-index documents for it to take effect.
Matches in longer fields are less precise, so implementations of this method usually set
smaller values when state.getLength() is large, and larger values when
state.getLength() is small.
Note that for a given term-document frequency, greater unsigned norms must produce scores
that are lower or equal, ie. for two encoded norms n1 and n2 so that
Long.compareUnsigned(n1, n2) > 0 then SimScorer.score(freq, n1) <=
SimScorer.score(freq, n2) for any legal freq.
0 is not a legal norm, so 1 is the norm that produces the highest scores.
| name | type | description |
|---|---|---|
| state | org.apache.lucene.index.FieldInvertState | accumulated state of term processing for this field |
Returns: computed norm value
getDiscountOverlaps() · also: get_discount_overlapsReturns true if overlap tokens are discounted from the document's length.
scorer(float boost, org.apache.lucene.search.CollectionStatistics collectionStats, org.apache.lucene.search.TermStatistics[] termStats)toString()Returns the name of the LM method. If a custom collection model strategy is used, its name is included as well.