我正在使用kafkasout将来自kafka队列的消息读取到风暴拓扑中,它在过去整整一周都运行良好,但从昨天开始,它给了我下面的错误,说内存不足。我已经检查了代码中所有与内存相关的参数和内存泄漏,但一切似乎都很好,
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) ~[na:1.7.0_45]
at java.lang.Thread.start(Thread.java:713) ~[na:1.7.0_45]
at backtype.storm.util$async_loop.doInvoke(util.clj:421) ~[storm-core-0.9.0-rc3.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:559) [clojure-1.4.0.jar:na]
at backtype.storm.daemon.executor$eval9473$fn__9474.invoke(executor.clj:732) ~[na:na]
at clojure.lang.MultiFn.invoke(MultiFn.java:167) ~[clojure-1.4.0.jar:na]
at backtype.storm.daemon.executor$mk_executor$fn__9342.invoke(executor.clj:317) ~[na:na]
at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:316) [na:na]
at backtype.storm.daemon.worker$eval9968$exec_fn__6466__auto____9969$iter__9974__9978$fn__9979. invoke(worker.clj:360) [na:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) [clojure-1.4.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) [clojure-1.4.0.jar:na]
at clojure.lang.Cons.next(Cons.java:39) [clojure-1.4.0.jar:na]
at clojure.lang.LazySeq.next(LazySeq.java:92) [clojure-1.4.0.jar:na]
at clojure.lang.RT.next(RT.java:587) [clojure-1.4.0.jar:na]
at clojure.core$next.invoke(core.clj:64) [na:na]
at clojure.core$dorun.invoke(core.clj:2726) [na:na]
at clojure.core$doall.invoke(core.clj:2741) [na:na]
at backtype.storm.daemon.worker$eval9968$exec_fn__6466__auto____9969.invoke(worker.clj:360) [na:na]
at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
at clojure.core$apply.invoke(core.clj:601) [na:na]
at backtype.storm.daemon.worker$eval9968$mk_worker__10024.doInvoke(worker.clj:329) [na:na]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) [na:na]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.0-rc3.jar:na]
2014-04-11 13:56:11 b.s.util [ERROR] Async loop died!
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) [na:1.7.0_45]
at java.lang.Thread.start(Thread.java:713) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[na:1.7.0_45]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_45]
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) ~[na:1.7.0_45]
at com.netflix.curator.framework.state.ConnectionStateManager.start(ConnectionStateManager.java:83) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:168) ~[curator-framework-1.0.1.jar:na]
at storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:35) ~[stormjar.jar:na]
at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:23) ~[stormjar.jar:na]
at storm.kafka.trident.KafkaUtils.makeBrokerReader(KafkaUtils.java:26) ~[stormjar.jar:na]
at storm.kafka.KafkaSpout.open(KafkaSpout.java:67) ~[stormjar.jar:na]
at backtype.storm.daemon.executor$eval9403$fn__9404$fn__9419.invoke(executor.clj:504) ~[na:na]
at backtype.storm.util$async_loop$fn__5748.invoke(util.clj:401) ~[na:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
2014-04-11 13:56:11 b.s.d.executor [ERROR]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method) [na:1.7.0_45]
at java.lang.Thread.start(Thread.java:713) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949) ~[na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360) ~[na:1.7.0_45]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132) ~[na:1.7.0_45]
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:641) ~[na:1.7.0_45]
at com.netflix.curator.framework.state.ConnectionStateManager.start(ConnectionStateManager.java:83) ~[curator-framework-1.0.1.jar:na]
at com.netflix.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:168) ~[curator-framework-1.0.1.jar:na]
at storm.kafka.DynamicBrokersReader.<init>(DynamicBrokersReader.java:35) ~[stormjar.jar:na]
at storm.kafka.trident.ZkBrokerReader.<init>(ZkBrokerReader.java:23) ~[stormjar.jar:na]
at storm.kafka.trident.KafkaUtils.makeBrokerReader(KafkaUtils.java:26) ~[stormjar.jar:na]
at storm.kafka.KafkaSpout.open(KafkaSpout.java:67) ~[stormjar.jar:na]
at backtype.storm.daemon.executor$eval9403$fn__9404$fn__9419.invoke(executor.clj:504) ~[na:na]
at backtype.storm.util$async_loop$fn__5748.invoke(util.clj:401) ~[na:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
2014-04-11 13:56:11 b.s.util [INFO] Halting process: ("Worker died")
2014-04-11 13:56:11 b.s.util [INFO] Halting process: ("Worker died")
我的风暴群参数是:nimbus.childopts-Xmx1024mworker.childopts-Xmx768mdrpc childopts-Xmx768m
无法创建新的本机线程与堆无关,这实际上是来自操作系统的代理消息,操作系统无法分配更多的本机内存来创建操作系统级别的新线程和/或进程。