Datastax DSE - 删除大量行导致"Insert command failed after 4 attempts"异常



我们有两个DSE节点(3.2.4)用于开发目的。 Cassandra 密钥空间的复制因子设置为 2。 当在没有指定一致性级别(即:仲裁)的情况下通过查询执行 Solr 删除时,偶尔我们在删除大量行(大约 500K 左右)时会出现以下异常。 将一致性级别设置为 1 时,还观察到相同的异常(删除大约 100 万行时)。 我想知道是否有解决方法。 例外情况如下:

 WARN [http-8983-1] 2014-03-27 09:21:59,008 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 1 out of 4 with CL ONE...
 WARN [http-8983-1] 2014-03-27 09:22:09,109 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 2 out of 4 with CL ONE...
 WARN [http-8983-1] 2014-03-27 09:22:19,211 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 3 out of 4 with CL ONE...
 WARN [http-8983-1] 2014-03-27 09:22:29,312 CassandraUtil.java (line 44) Error Operation timed out - received only 0 responses. on attempt 4 out of 4 with CL ONE...
ERROR [http-8983-1] 2014-03-27 09:22:29,415 SolrException.java (line 124) org.apache.solr.common.SolrException: java.lang.RuntimeException: Insert command failed after 4 attempts, source exception follows.
        at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.deleteByQuery(CassandraDirectUpdateHandler.java:236)
        at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:80)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:899)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:757)
        at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1841)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:669)
        at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.execute(CassandraDispatchFilter.java:165)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
        at com.datastax.bdp.search.solr.servlet.CassandraDispatchFilter.doFilter(CassandraDispatchFilter.java:137)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.cassandra.audit.SolrHttpAuditLogFilter.doFilter(SolrHttpAuditLogFilter.java:194)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.search.solr.auth.CassandraAuthorizationFilter.doFilter(CassandraAuthorizationFilter.java:94)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at com.datastax.bdp.search.solr.auth.DseAuthenticationFilter.doFilter(DseAuthenticationFilter.java:102)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Insert command failed after 4 attempts, source exception follows.
        at com.datastax.bdp.util.CassandraUtil.robustInsert(CassandraUtil.java:56)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.doDeletes(ThriftCassandraRowWriter.java:189)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.access$100(ThriftCassandraRowWriter.java:39)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter$DeletingHitCollector.collect(ThriftCassandraRowWriter.java:241)
        at org.apache.lucene.search.Scorer.score(Scorer.java:65)
        at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:245)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:612)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
        at com.datastax.bdp.search.solr.ThriftCassandraRowWriter.deleteByQuery(ThriftCassandraRowWriter.java:168)
        at com.datastax.bdp.search.solr.handler.update.CassandraDirectUpdateHandler.deleteByQuery(CassandraDirectUpdateHandler.java:217)
        ... 38 more
Caused by: org.apache.cassandra.exceptions.WriteTimeoutException: Operation timed out - received only 0 responses.
        at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:91)
        at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:198)
        at com.datastax.bdp.util.CassandraUtil.robustInsert(CassandraUtil.java:39)
        ... 47 more

感谢您报告此问题。我们正在研究它并考虑改进。

目前,我所能建议的是,如果操作失败,请重试该操作,稍有延迟,因为问题可能只是群集繁忙。

您可能还需要考虑较小的删除,使用选择较少文档/行的较窄查询。

我将检查您是否可以合理调整超时以处理群集延迟问题。

编辑:您可以在cassandra.yaml中使用write_request_timeout_in_ms提高超时

最新更新