HandleHttpRequest 失败,"rate of the dataflow is exceeding the provenance recording rate"时出现SERVICE_UNA



我有一个使用jMeter的批量测试,它使用HandleHttpRequest向处理器NIFI发送几个HTTP请求(GET),并发送到Topic Kafka。

问题是StandardHTTPContextMap返回SERVICE_UNAVAILABLE错误,当数据流的速率超过出处记录速率时,似乎会发生这种情况,但我不确定。

有人知道吗?我删除了一个部分日志:

2016-05-05 15:12:14,064 WARN [Timer-Driven Process Thread-7] 
o.a.n.p.PersistentProvenanceRepository The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate. Currently, there are 96 journal files (533328812 bytes) and threshold for blocking is 80 (1181116006 bytes)
2016-05-05 15:12:20,310 INFO [Provenance Repository Rollover Thread-2] 
o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (46096 records) into single Provenance Log File ./provenance_repository/8913710.prov in 43254 milliseconds
2016-05-05 15:12:20,314 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 65422 records
2016-05-05 15:12:20,398 INFO [Timer-Driven Process Thread-7]   
o.a.n.p.PersistentProvenanceRepository Provenance Repository has now caught up with rolling over journal files. Current number of journal files to be rolled over is 80
2016-05-05 15:12:20,399 INFO [Timer-Driven Process Thread-7] 
o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 9190418
2016-05-05 15:12:21,422 INFO [qtp1693512967-121] 
o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=3858f0ad-b165-427b-a460-67fbf7cff0d8] Sending back a SERVICE_UNAVAILABLE response to 172.26.60.27; request was GET 172.26.60.27

您的分析是正确的,您看到的HTTP响应来自HttpContextMap[1]。特别是"请求过期"属性。当请求在地图中超过配置的数量时,它将自动回复SERVICE_UNAVAILABLE。

我猜你的问题是,NiFi处理你提交的所有请求的时间太长,导致Provence Repo强制展期,这是一个"停止世界"事件。因此,您停止处理任何数据6秒钟(导致请求过期)。

假设你不想只接受随机的6秒"停止世界"事件,并且对你的流量或配置一无所知,那么你基本上需要缩放或调整你的流量。有几个选项:

  • 缩放到更大的节点或更多节点
  • 处理较大的FlowFiles而不是许多FlowFiles(有助于加快Provence)
  • 将出处回购推送到自己的磁盘/推送到多个磁盘

[1]https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.http.StandardHttpContextMap/index.html

相关内容

最新更新