Accumulo的createtable命令卡住并且无法创建表

Question

我试图使用createtable命令在Accumulo中创建一个表，但发现它被卡住了。我等了大约20分钟才取消createtable命令。

createtable test_table

我有一台主机和两台平板电脑服务器，发现我的主机和其中一台平板电脑都死了。我无法远程登录到特定平板电脑服务器的9997端口，甚至无法远程登录29999端口（accumulo-site.xml中的master.port.client）。当我看到死服务器的tserver日志时，我看到了以下条目。

2016-05-10 02:12:07,456 [zookeeper.DistributedWorkQueue] INFO : Got unexpected z
ookeeper event: None for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/recovery
2016-05-10 02:12:23,883 [zookeeper.ZooCache] WARN : Saw (possibly) transient exc
eption communicating with ZooKeeper, will retry
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/tables
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.accumulo.fate.zookeeper.ZooCache$1.run(ZooCache.java:210)
        at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
        at org.apache.accumulo.fate.zookeeper.ZooCache.getChildren(ZooCache.java
:221)
        at org.apache.accumulo.core.client.impl.Tables.exists(Tables.java:142)
        at org.apache.accumulo.server.tabletserver.LargestFirstMemoryManager.tab
leExists(LargestFirstMemoryManager.java:149)
        at org.apache.accumulo.server.tabletserver.LargestFirstMemoryManager.get
MemoryManagementActions(LargestFirstMemoryManager.java:175)
        at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagem
entFramework.manageMemory(TabletServerResourceManager.java:408)
        at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagem
entFramework.access$400(TabletServerResourceManager.java:318)
        at org.apache.accumulo.tserver.TabletServerResourceManager$MemoryManagem
entFramework$2.run(TabletServerResourceManager.java:346)
        at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.jav
a:35)
        at java.lang.Thread.run(Thread.java:745)
2016-05-10 02:12:23,884 [zookeeper.ZooCache] WARN : Saw (possibly) transient exc
eption communicating with ZooKeeper, will retry
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/tables/!0/con
f/table.classpath.context
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.accumulo.fate.zookeeper.ZooCache$2.run(ZooCache.java:264)
        at org.apache.accumulo.fate.zookeeper.ZooCache.retry(ZooCache.java:162)
        at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:289)
        at org.apache.accumulo.fate.zookeeper.ZooCache.get(ZooCache.java:238)
        at org.apache.accumulo.server.conf.ZooCachePropertyAccessor.get(ZooCache
PropertyAccessor.java:117)
        at org.apache.accumulo.server.conf.ZooCachePropertyAccessor.get(ZooCache
PropertyAccessor.java:103)
        at org.apache.accumulo.server.conf.TableConfiguration.get(TableConfigura
tion.java:99)
        at org.apache.accumulo.tserver.constraints.ConstraintChecker.classLoader
Changed(ConstraintChecker.java:93)
        at org.apache.accumulo.tserver.tablet.Tablet.checkConstraints(Tablet.jav
a:1225)
        at org.apache.accumulo.tserver.TabletServer$8.run(TabletServer.java:2848
)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51
1)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-05-10 02:12:23,887 [zookeeper.ZooReader] WARN : Saw (possibly) transient ex
ception communicating with ZooKeeper
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /accumulo/be4f66be-1508-4314-9bff-888b56d9b0ce/tservers/accu
mulo.tablet.2:9997
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
        at org.apache.accumulo.fate.zookeeper.ZooReader.getStatus(ZooReader.java
:132)
        at org.apache.accumulo.fate.zookeeper.ZooLock.process(ZooLock.java:383)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.j
ava:522)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
2016-05-10 02:12:24,252 [watcher.MonitorLog4jWatcher] INFO : Changing monitor lo
g4j address to accumulo.master:4560
2016-05-10 02:12:24,252 [watcher.MonitorLog4jWatcher] INFO : Enabled log-forward
ing

甚至主服务器的日志也有相同的堆栈竞争。我的动物园管理员正在奔跑。

起初，我以为是磁盘问题。也许没有空间。但事实并非如此。我在accumulo instance.volumes上运行了fsck，它返回了健康状态。

有人知道到底发生了什么吗？如果可能的话，如何避免？

EDIT：甚至tracer_accumulo.master.log也有相同的stacktrace。

Answer 1

ZooKeeper客户端内的线程没有在必要的时间内（默认情况下为30s）运行以维护ZooKeeper-客户端和服务器之间处于内存状态的会话时，就会发生ZooKeepers会话过期。对此没有单一的解释，但有许多常见的罪魁祸首：

1. JVM垃圾收集在客户端暂停。如果Accumulo遇到暂停，则应记录警告
2. CPU时间不足。如果主机本身负担过重，Accumulo可能没有周期来及时运行所需的所有任务
3. 缺少套接字/文件句柄，Accumulo可能正在尝试连接到ZooKeeper，但无法打开新连接
4. ZooKeeper可能会限制连接速率以防止拒绝服务。请检查zookeeper日志中有关从特定IP删除/拒绝新连接的错误，如果您看到这些错误，请考虑在zoo.cfg中增加maxClientCnxns