Apache Cassandra不需要的提示文件存在于提示文件夹中

Question

在Apache Cassandra 3.11.1上，在Cassandra退役期间，我注意到节点试图发送提示。我检查了提示文件夹，发现提示非常旧或已经删除的节点。我决定删除它们，这样它们就不是退役过程的一部分，但是在退役节点后，该过程失败，节点被卡在UL状态。

异常堆栈:

ERROR [HintsDispatcher:3] 2022-12-12 17:58:39,364 CassandraDaemon.java:228 - Exception in thread Thread[HintsDispatcher:3,1,RMI Runtime]
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/xxxx-1670867104313-1.hints
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.ChecksummedDataInput.open(ChecksummedDataInput.java:77) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsReader.open(HintsReader.java:78) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatcher.create(HintsDispatcher.java:73) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:273) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:260) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:238) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:217) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_131]
at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_131]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/0891d19f-7ba9-4fc6-973c-79f98253cf4e-1670867104313-1.hints
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.8.0_131]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 25 common frames omitted

最后一个错误:

ERROR [RMI TCP Connection(350)-127.0.0.1] 2022-12-12 18:05:33,190 StorageService.java:3954 - Error while decommissioning node 
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/xxxx-1670867104313-1.hints
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.io.util.ChannelProxy.<init>(ChannelProxy.java:66) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.ChecksummedDataInput.open(ChecksummedDataInput.java:77) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsReader.open(HintsReader.java:78) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatcher.create(HintsDispatcher.java:73) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:273) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:260) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:238) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:217) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_131]
at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_131]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_131]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_131]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.transfer(HintsDispatchExecutor.java:186) ~[apache-cassandra-3.11.1.jar:3.11.1]
at org.apache.cassandra.hints.HintsDispatchExecutor$TransferHintsTask.run(HintsDispatchExecutor.java:159) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_131]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) ~[apache-cassandra-3.11.1.jar:3.11.1]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.nio.file.NoSuchFileException: /var/lib/cassandra/hints/0891d19f-7ba9-4fc6-973c-79f98253cf4e-1670867104313-1.hints
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[na:1.8.0_131]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[na:1.8.0_131]
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:287) ~[na:1.8.0_131]
at java.nio.channels.FileChannel.open(FileChannel.java:335) ~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ~[apache-cassandra-3.11.1.jar:3.11.1]
... 25 common frames omitted

我怎样才能避免这种情况?

我尝试了nodetool truncatehints，但它在节点上没有任何作用。

Answer 1

此时，我将在其上运行nodetool removenode。removenode进程将:

• 清除卡住的DL/UL节点
• 将数据从其他副本移动到新的令牌范围。

如果所有数据都已经从离开节点迁移了，会发生什么?removenode命令会再次流并创建重复数据吗?

是的，它会再次流式传输，但它不会创建"重复数据"，这与就地写入的方式不同。

所以清理将清除这些重新流的数据?

。流发生在SSTable文件级别。因此，从本质上讲，在decomm/remove过程中，一个全新的文件正在流式传输。如果已经存在重复的、就地更新的数据，它仍然会在其新节点上。如果在分区级别上只有一个写操作，那么就只有这个了。如果它已经存在，这个进程将简单地把一个文件写到另一个文件上，Cassandra甚至永远不会知道它的存在。

添加或删除节点时，触发令牌范围计算。这意味着一些节点最终包含它们不再负责的数据(因为另一个节点现在负责)。运行nodetool cleanup将删除以下数据:

1. 已经在其他地方流了
2. 保留在节点上我永远不会使用它。