Just enough Information Retrieval theory to find your way around Apache Lucy.
Lucy uses some terminology from the field of information retrieval which may be unfamiliar to many users. “Document” and “term” mean pretty much what you’d expect them to, but others such as “posting” and “inverted index” need a formal introduction:
Since Lucy is a practical implementation of IR theory, it loads these abstract, distilled definitions down with useful traits. For instance, a “posting” in its most rarefied form is simply a term-document pairing; in Lucy, the class MatchPosting fills this role. However, by associating additional information with a posting like the number of times the term occurs in the document, we can turn it into a ScorePosting, making it possible to rank documents by relevance rather than just list documents which happen to match in no particular order.
Lucy uses a variant of the well-established “Term Frequency / Inverse Document Frequency” weighting scheme. A thorough treatment of TF/IDF is too ambitious for our present purposes, but in a nutshell, it means that…
in a search for
skate park, documents which score well for the
comparatively rare term
skate will rank higher than documents which score
well for the more common term
a 10-word text which has one occurrence each of both
rank higher than a 1000-word text which also contains one occurrence of each.
A web search for “tf idf” will turn up many excellent explanations of the algorithm.
Copyright © 2010-2015 The Apache Software Foundation, Licensed under the
Apache License, Version 2.0.
Apache Lucy, Lucy, Apache, the Apache feather logo, and the Apache Lucy project logo are trademarks of The Apache Software Foundation. All other marks mentioned may be trademarks or registered trademarks of their respective owners.