许多PDF索引，但在Te Solr-UI中只返回

Question

我遵循此处的示例，以索引目录中的所有PDF。该过程似乎运行良好，但是最后，当我进入Solr-UI并单击"执行查询"（Q =*：*）时，我只得到一个条目。

我是否错过了我的代码中的东西？

...
String[] files = documentDir.list();
if (files != null)
{
  for (String document : files)
  {       
    ContentHandler textHandler = new BodyContentHandler();
    Metadata metadata = new Metadata();
    ParseContext context = new ParseContext();
    AutoDetectParser autoDetectParser = new AutoDetectParser();
    InputStream inputStream = null;
    try
    {
      inputStream = new FileInputStream(new File(documentDir, document));
      autoDetectParser.parse(inputStream, textHandler, metadata, context);
      SolrInputDocument doc = new SolrInputDocument();
      doc.addField("id", document);
      String content = textHandler.toString();
      if (content != null)
      {
        doc.addField("fullText", content);
      }
      UpdateResponse resp = server.add(doc, 1);
      server.commit(true, true, true);
      if (resp.getStatus() != 0)
      {
        throw new IDSystemException(LOG, "Document could not be indexed. Status returned: " + resp.getStatus());
      }
    }
    catch (FileNotFoundException fnfe)
    {
      throw new IDSystemException(LOG, fnfe.getMessage(), fnfe);
    }
    catch (IOException ioe)
    {
      throw new IDSystemException(LOG, ioe.getMessage(), ioe);
    }
    catch (SAXException se)
    {
      throw new IDSystemException(LOG, se.getMessage(), se);
    }
    catch (TikaException te)
    {
      throw new IDSystemException(LOG, te.getMessage(), te);
    }
    catch (SolrServerException sse)
    {
      throw new IDSystemException(LOG, sse.getMessage(), sse);
    }
    finally
    {
      if (inputStream != null)
      {
        try
        {
          inputStream.close();
        }
        catch (IOException ioe)
        {
          throw new IDSystemException(LOG, ioe.getMessage(), ioe);
        }
      }
    }
    ...

Answer 1

i在solrconfig.xml中绑定到" uid"的" signature field"，因此uid始终是相同的。现在，我为" SignatureField"定义了一个新字段，它可以工作！

之前：

...
<updateRequestProcessorChain name="deduplication">
    <processor
        class="org.apache.solr.update.processor.SignatureUpdateProcessorFactory">
        <bool name="overwriteDupes">false</bool>
        <str name="signatureField">uid</str>  <---------------------
        <bool name="enabled">true</bool>
        <str name="fields">content</str>
        <str name="minTokenLen">10</str>
        <str name="quantRate">.2</str>
        <str name="signatureClass">solr.update.processor.TextProfileSignature</str>
    </processor>
    <processor class="solr.LogUpdateProcessorFactory" />
    <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
...

...
<fields>
    <field name="uid" type="string" indexed="true" stored="true" multiValued="false" />
    <dynamicField name="ignored_*" type="ignored" multiValued="true" indexed="false" stored="fasle" />
    <field name="id" type="string" indexed="true" stored="true" multiValued="false" />
    <field name="fullText" indexed="true" type="text" multiValued="true" />
</fields>
<uniqueKey>uid</uniqueKey>
...

之后：

...
<updateRequestProcessorChain name="deduplication">
    <processor
        class="org.apache.solr.update.processor.SignatureUpdateProcessorFactory">
        <bool name="overwriteDupes">false</bool>
        <str name="signatureField">signatureField</str>  <---------------------
        <bool name="enabled">true</bool>
        <str name="fields">content</str>
        <str name="minTokenLen">10</str>
        <str name="quantRate">.2</str>
        <str name="signatureClass">solr.update.processor.TextProfileSignature</str>
    </processor>
    <processor class="solr.LogUpdateProcessorFactory" />
    <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>
...

...
<fields>
    <field name="uid" type="string" indexed="true" stored="true" multiValued="false" />
    <field name="signatureField" type="string" indexed="true" stored="true" multiValued="false" />  <----------------------------------
    <dynamicField name="ignored_*" type="ignored" multiValued="true" indexed="false" stored="fasle" />
    <field name="id" type="string" indexed="true" stored="true" multiValued="false" />
    <field name="fullText" indexed="true" type="text" multiValued="true" />
</fields>
<uniqueKey>uid</uniqueKey>
...

Answer 2

也许可以使用server.add检查您的行。为什么您有1个作为第二个参数？

您是否考虑过使用Nutch，它会为您爬行和索引吗？