线程 "main" com.datastax.driver.core.exceptions.ServerError(英语:服务器上的服务器端发生意外错误)



在尝试使用分页支持从 cassandra 获取大约 500000 条记录时,我有时会观察到以下异常。请注意,这是一个间歇性异常,每次运行程序时都看不到。

卡桑德拉版本 : 3.10 数据税驱动程序:3.1.0

Statement statement = new SimpleStatement("select * from mytable");
rs = session.execute(stmt);
for (Row row : rs) {
if (rs.getAvailableWithoutFetching() == 100) {
rs.fetchMoreResults();
}
// Do something with row 
}

有时在获取大约 200000 条记录后,程序会失败,并出现以下异常:

Exception in thread "main" com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on myhost:9042: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:63)
at com.datastax.driver.core.exceptions.ServerError.copy(ServerError.java:25)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.ArrayBackedResultSet$MultiPage.prepareNextRow(ArrayBackedResultSet.java:312)
at com.datastax.driver.core.ArrayBackedResultSet$MultiPage.isExhausted(ArrayBackedResultSet.java:269)
at com.datastax.driver.core.ArrayBackedResultSet$1.hasNext(ArrayBackedResultSet.java:143)
Caused by: com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on myhost:9042: com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.datastax.driver.core.Responses$Error.asException(Responses.java:108)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:500)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1012)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:935)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:321)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1280)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:328)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:890)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:564)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:505)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:419)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:391)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:748)

卡桑德拉日志 :

WARN  [Native-Transport-Requests-1] 2018-01-09 10:07:31,832 CassandraRoleManager.java:96 - An invalid value has been detected in the roles table for role myuser. If you are unable to login, you may need to disable authentication and confirm that values in that table are accurate
ERROR [Native-Transport-Requests-1] 2018-01-09 10:07:31,833 QueryMessage.java:129 - Unexpected error during query
com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 26 common frames omitted
Caused by: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:281) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 40 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88) ~[apache-cassandra-3.10.jar:3.10]
... 50 common frames omitted
ERROR [Native-Transport-Requests-1] 2018-01-09 10:07:31,834 ErrorMessage.java:384 - Unexpected exception during request
com.google.common.util.concurrent.UncheckedExecutionException: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.authorize(ClientState.java:419) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:352) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:329) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:316) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:300) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:221) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:214) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:233) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410) [apache-cassandra-3.10.jar:3.10]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357) [netty-all-4.0.39.Final.jar:4.0.39.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_152]
at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_152]
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:81) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.PermissionsCache.lambda$new$0(PermissionsCache.java:37) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 26 common frames omitted
Caused by: java.lang.RuntimeException: Invalid metadata has been detected for role myuser
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:99) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:82) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:528) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:503) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:281) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.10.jar:3.10]
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
... 40 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.cassandra.cql3.UntypedResultSet$Row.getBoolean(UntypedResultSet.java:273) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.auth.CassandraRoleManager$1.apply(CassandraRoleManager.java:88) ~[apache-cassandra-3.10.jar:3.10]
... 50 common frames omitted
INFO  [Native-Transport-Requests-1] 2018-01-09 10:08:06,156 Message.java:617 - Unexpected exception during request; channel = [id: 0xbbaa062e, L:/127.0.0.1:9042 ! R:/127.0.0.1:9042]
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer
at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.39.Final.jar:4.0.39.Final]

我也遇到了同样的问题。当跨节点的 system_auth.roles 表不同步时,似乎会出现此错误。为了解决此错误,我执行了以下步骤并得到了解决。

  1. 从 system_auth.roles 中选择 *,其中角色 = "抛出错误的角色";
  2. 节点工具修复system_auth; 一个接一个地 #on 所有节点
  3. 一致性法定人数;
  4. 从system_auth.roles中选择*;#Read 所有角色一次
  5. 列出角色;

最新更新