标记结构化查询同性恋

Question

我正在研究一个应用程序，其中最多需要在2小时内获取100万个文件

我们使用Java客户端API和结构化查询来执行搜索。但是，查询仍然很慢。

代码如下：

def fetchPostMessages(dbParam: DbParam): Page = {
  val queryManager = dbClient.newQueryManager()
  val sqb: StructuredQueryBuilder = queryManager.newStructuredQueryBuilder()
  log.info(s "Fetching post messages from database for params: {}", dbParam)
  val modifiedQueryDef = dbParam.param.map {
    param => {
      sqb.and(
        sqb.word(sqb.jsonProperty(status), toBeReported),
        sqb.word(sqb.jsonProperty(dataCategory), "dataCategory1"),
        sqb.range(sqb.jsonProperty(creationDate), marklogicDateFormat.name, Operator.LE, DateUtil.printFpmlDateTime(param.messagesTime)))
    }
  }.getOrElse(sqb.and(sqb.word(sqb.jsonProperty(status.name), toBeReported.name)))
  modifiedQueryDef.setCollections(XmlConstants.ItracMessageTypes.OUTPUT_MESSAGE.name)
  modifiedQueryDef.setOptionsName(sortOption)
  search(modifiedQueryDef, dbParam.pageNum, dbParam.batchSize)
}

private def search(queryDef: QueryDefinition, startIndex: Int, batchSize: Int): Page = {
  val dataList: ListBuffer[Document] = new ListBuffer()
  val jsonDocManager = dbClient.newJSONDocumentManager()
  jsonDocManager.setMetadataCategories(Metadata.ALL)
  jsonDocManager.setPageLength(
    if (batchSize < pageLength) batchSize
    else pageLength)
  val documentPage = jsonDocManager.search(queryDef, startIndex);
  dataList.++ = (extractContent(documentPage))
  val totalSize = documentPage.getTotalSize
  log.info(s "Total documents to be reported : ${totalSize}")
  var pageSize = documentPage.getPageSize
  while (pageSize < batchSize && pageSize <= totalSize) {
    if (batchSize - pageSize < pageSize)
      jsonDocManager.setPageLength(batchSize - pageSize)
    var newDocPage = jsonDocManager.search(queryDef, pageSize + 1)
    dataList.++ = (extractContent(newDocPage))
    pageSize = pageSize + newDocPage.getPageSize
  }
  log.info("Total messages fetched are : {}", dataList.size)
  Page(startIndex, totalSize - batchSize, dataList.to[collection.immutable.Seq])
}

排序选项是：

<search:options xmlns:search="http://marklogic.com/appservices/search">
  <search:sort-order type="xs:string" direction="ascending">
    <search:json-property>subdomLvl1</search:json-property>
  </search:sort-order>
  <search:sort-order type="xs:string" direction="ascending">
    <search:json-property>trdId</search:json-property>
  </search:sort-order>
  <search:sort-order type="xs:string" direction="ascending">
    <search:json-property>validStartDate</search:json-property>
  </search:sort-order>
  <search:sort-order type="xs:string" direction="ascending">
    <search:json-property>ver</search:json-property>
  </search:sort-order>
  <search:sort-order type="xs:string" direction="ascending">
    <search:json-property>reportStatus</search:json-property>
  </search:sort-order>
</search:options>

数据库上的索引如下：

元素范围索引已打开 - 状态，dataCategory和createDate以及各种选项

Answer 1

如果该过程不需要文档元数据，请考虑使用jsondocmanager.clearmetadataCategories（）而不是jsondocmanager.setmetadataTaCategories（metadata.all）配置。该方法将减少服务器和客户端的工作，并减少传输的数据。

可以通过测试newDocpage.hasnextpage（）来简化循环 - 请参阅：

http://docs.marklogic.com/guide/java/bulk#id_21619

客户不能在单个列表中累积所有百万个文档，而是可以将文件传输到消费过程中吗？那肯定会改善吞吐量。

您还可以考虑使用数据移动SDK在多个线程中读取文档：

http://docs.marklogic.com/guide/java/data-movement#id_60613

http://docs.marklogic.com/javadoc/client/com/marklogic/client/datamovement/querybatcher.html

希望有帮助，