我有一个程序，它接受输入查询，并根据TFIDF得分对类似文档进行排名。问题是，我想添加一些关键词，并将它们视为"输入"。这些关键字对于每个查询都是不同的。
例如，如果查询是"Logic Based Knowledge Representation"，则单词如下：
Level 0 keywords: [logic, base, knowledg, represent]
Level 1 keywords: [tempor, modal, logic, resolut, method, decis, problem,
reason, revis, hybrid, represent]
Level 2 keywords: [classif, queri, process, techniqu, candid, semant, data,
model, knowledg, base, commun, softwar, engin, subsumpt,
kl, undecid, classic, structur, object, field]
我想以不同的方式对待分数，例如，对于文档中等于0级单词的术语，我想将分数乘以1。对于文档中等于一级单词的术语，将分数乘以0.8。最后，对于文档中等于第二级单词的术语，将分数乘以0.64。
我的目的是扩展输入查询，但也要确保包含更多级别0关键字的文档被视为更重要，而包含级别1和2关键字的文档则更少（即使扩展了输入）。我还没有把它包括在我的程序中。到目前为止，我的程序只计算所有文档的TFIDF分数，并对结果进行排名：
public class Ranking{
private static int maxHits = 2000000;
public static void main(String[] args) throws Exception {
System.out.println("Enter your paper title: ");
BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
String paperTitle = null;
paperTitle = br.readLine();
// CitedKeywords ckeywords = new CitedKeywords();
// ckeywords.readDataBase(paperTitle);
String querystr = args.length > 0 ? args[0] :paperTitle;
StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_42);
Query q = new QueryParser(Version.LUCENE_42, "title", analyzer)
.parse(querystr);
IndexReader reader = DirectoryReader.open(
FSDirectory.open(
new File("E:/Lucene/new_bigdataset_index")));
IndexSearcher searcher = new IndexSearcher(reader);
VSMSimilarity vsmSimiliarty = new VSMSimilarity();
searcher.setSimilarity(vsmSimiliarty);
TopDocs hits = searcher.search(q, maxHits);
ScoreDoc[] scoreDocs = hits.scoreDocs;
PrintWriter writer = new PrintWriter("E:/Lucene/result/1.txt", "UTF-8");
int counter = 0;
for (int n = 0; n < scoreDocs.length; ++n) {
ScoreDoc sd = scoreDocs[n];
float score = sd.score;
int docId = sd.doc;
Document d = searcher.doc(docId);
String fileName = d.get("title");
String year = d.get("pub_year");
String paperkey = d.get("paperkey");
System.out.printf("%s,%s,%s,%4.3fn", paperkey, fileName, year, score);
writer.printf("%s,%s,%s,%4.3fn", paperkey, fileName, year, score);
++counter;
}
writer.close();
}
}
--
import org.apache.lucene.index.FieldInvertState;
import org.apache.lucene.search.similarities.DefaultSimilarity;
public class VSMSimilarity extends DefaultSimilarity{
// Weighting codes
public boolean doBasic = true; // Basic tf-idf
public boolean doSublinear = false; // Sublinear tf-idf
public boolean doBoolean = false; // Boolean
//Scoring codes
public boolean doCosine = true;
public boolean doOverlap = false;
private static final long serialVersionUID = 4697609598242172599L;
// term frequency in document =
// measure of how often a term appears in the document
public float tf(int freq) {
// Sublinear tf weighting. Equation taken from [1], pg 127, eq 6.13.
if (doSublinear){
if (freq > 0){
return 1 + (float)Math.log(freq);
} else {
return 0;
}
} else if (doBoolean){
return 1;
}
// else: doBasic
// The default behaviour of Lucene is sqrt(freq),
// but we are implementing the basic VSM model
return freq;
}
// inverse document frequency =
// measure of how often the term appears across the index
public float idf(int docFreq, int numDocs) {
if (doBoolean || doOverlap){
return 1;
}
// The default behaviour of Lucene is
// 1 + log (numDocs/(docFreq+1)),
// which is what we want (default VSM model)
return super.idf(docFreq, numDocs);
}
// normalization factor so that queries can be compared
public float queryNorm(float sumOfSquaredWeights){
if (doOverlap){
return 1;
} else if (doCosine){
return super.queryNorm(sumOfSquaredWeights);
}
// else: can't get here
return super.queryNorm(sumOfSquaredWeights);
}
// number of terms in the query that were found in the document
public float coord(int overlap, int maxOverlap) {
if (doOverlap){
return 1;
} else if (doCosine){
return 1;
}
// else: can't get here
return super.coord(overlap, maxOverlap);
}
// Note: this happens an index time, which we don't take advantage of
// (too many indices!)
public float computeNorm(String fieldName, FieldInvertState state){
if (doOverlap){
return 1;
} else if (doCosine){
return super.computeNorm(state);
}
// else: can't get here
return super.computeNorm(state);
}
}
以下是我当前程序的输出示例（没有提升分数）：
3086,Logic Based Knowledge Representation.,1999,5.165
33586,A Logic for the Representation of Spatial Knowledge.,1991,4.663
328937,Logic Programming for Knowledge Representation.,2007,4.663
219720,Logic for Knowledge Representation.,1984,4.663
487587,Knowledge Representation with Logic Programs.,1997,4.663
806195,Logic Programming as a Representation of Knowledge.,1983,4.663
806833,The Role of Logic in Knowledge Representation.,1983,4.663
744914,Knowledge Representation and Logic Programming.,2002,4.663
1113802,Knowledge Representation in Fuzzy Logic.,1989,4.663
984276,Logic Programming and Knowledge Representation.,1994,4.663
有人能告诉我如何为我上面提到的条件增加分数吗？Lucene提供这种功能吗？我可以将其集成到VSMSimilarity类中吗？
编辑：我在Lucene文档中发现了这一点：
public void setBoost(float b)
将此查询子句的提升设置为b。匹配此子句的文档将（除了正常权重外）其分数乘以b。
不幸的是，这似乎是文档级别分数的乘积。我想做一个学期水平的分数乘法，但我还没有找到方法。因此，如果文档包含级别0和级别1的单词，则只有级别1的术语会乘以0.8，例如
您可以使用Lucene术语boosts。
https://lucene.apache.org/core/5_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#Boosting_a_Term
像一样增强您的查询（假设OR是默认运算符）
logic base knowledge representation temporal^0.8 modal^0.8 classification^0.64...
并使用一个标准的模拟提供者。
PS：在您的示例中找到了LUCENE_42。这个功能几乎存在于Lucene的任何版本中（我记得它在2.4.9中出现过）