Jenkins 在执行器上使用 100% CPU 构建作业



一段时间以来,我们看到我们的 Jenkins 机器被固定在 100% CPU(或 200% 或 400%,取决于内核数量):

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                      
23376 ec2-user  20   0 4220m 100m    0 S 99.2  1.3 324945:21 java

即使在构建完成并且当前没有正在运行构建之后,这些 JVM 进程仍然存在。主节点和从属节点上的构建都会出现此问题。运行从属代理程序本身的 JVM 具有完全正常的 CPU 使用率。

一旦我终于能够获得线程转储,只有一个非系统线程可以运行并且不等待锁定:

"main" #1 prio=5 os_prio=0 tid=0x00007f0834008800 nid=0x5b51 runnable [0x00007f083abf3000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Harmonized.detach(MethodGraph.java:878)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store$Entry$Resolved.asNode(MethodGraph.java:1331)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default$Key$Store.asGraph(MethodGraph.java:1138)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$Default.compile(MethodGraph.java:507)
at net.bytebuddy.dynamic.scaffold.MethodGraph$Compiler$AbstractBase.compile(MethodGraph.java:423)
at net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:489)
at net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamqicTypeBuilder.make(SubclassDynamicTypeBuilder.java:153)
at net.bytebuddy.dynamic.DynamicType$Builder$AbstractBase$Delegator.make(DynamicType.java:2508)
at org.mockito.internal.creation.bytebuddy.MockBytecodeGenerator.generateMockClass(MockBytecodeGenerator.java:60)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.generate(CachingMockBytecodeGenerator.java:72)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator$CachedBytecodeGenerator.getOrGenerateMockClass(CachingMockBytecodeGenerator.java:64)
at org.mockito.internal.creation.bytebuddy.CachingMockBytecodeGenerator.get(CachingMockBytecodeGenerator.java:27)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createProxyClass(ByteBuddyMockMaker.java:54)
at org.mockito.internal.creation.bytebuddy.ByteBuddyMockMaker.createMock(ByteBuddyMockMaker.java:27)
at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:32)
at org.mockito.internal.MockitoCore.mock(MockitoCore.java:55)
at org.mockito.Mockito.mock(Mockito.java:1449)
at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:33)
at org.mockito.internal.configuration.MockAnnotationProcessor.process(MockAnnotationProcessor.java:16)
at org.mockito.internal.configuration.DefaultAnnotationEngine.createMockFor(DefaultAnnotationEngine.java:43)
at org.mockito.internal.configuration.DefaultAnnotationEngine.process(DefaultAnnotationEngine.java:66)
at org.mockito.internal.configuration.InjectingAnnotationEngine.processIndependentAnnotations(InjectingAnnotationEngine.java:71)
at org.mockito.internal.configuration.InjectingAnnotationEngine.process(InjectingAnnotationEngine.java:55)
at org.mockito.MockitoAnnotations.initMocks(MockitoAnnotations.java:108)
at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl$1.withBefores(JUnit45AndHigherRunnerImpl.java:27)
at org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:276)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.mockito.internal.runners.JUnit45AndHigherRunnerImpl.run(JUnit45AndHigherRunnerImpl.java:37)
at org.mockito.runners.MockitoJUnitRunner.run(MockitoJUnitRunner.java:62)
at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:59)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.executeTestSet(AbstractDirectoryTestSuite.java:115)
at org.apache.maven.surefire.suite.AbstractDirectoryTestSuite.execute(AbstractDirectoryTestSuite.java:102)
at org.apache.maven.surefire.Surefire.run(Surefire.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.maven.surefire.booter.SurefireBooter.runSuitesInProcess(SurefireBooter.java:350)
at org.apache.maven.surefire.booter.SurefireBooter.main(SurefireBooter.java:1021)

我连续进行了几次线程转储,这个线程几乎总是在完全相同的位置 - HashMap.put(HashMap.java:611) - 而JVM占用了100%的CPU。发生这种情况时,我没有机会附加调试器,但我只是想看看是否有人认识到这是一个已知的错误。对我来说,它看起来像某种热旋转的无限循环,试图一遍又一遍地将一些项目添加到哈希映射中。这可能与 Jenkins 有关,但在堆栈跟踪中出现的其他主要参与者显然是 Maven Surefire、JUnit、Mockito 和 ByteBuddy(很抱歉包含所有这些标签)。

我无法可靠地重现此问题,不幸的是,我也不知道我们的数百个构建中的哪一个留下了这些搞砸的 JVM。为了完整起见,环境是Jenkins 2.46.1,Maven 3.3.3,Surefire 2.19.1,JUnit 4.12,但不幸的是,一系列不同的Mockito(以及ByteBuddy)版本。我希望有人认识到这是所涉及的组件之一中的已知错误,并且可以建议解决方法......

你能用最新版本的 Mockito 重试吗?这似乎是我们用于模拟创建的新锁定机制的问题。以前,我们只对所有类型的使用一个锁,现在锁更细粒度。

您看到的堆栈跟踪是在模拟创建中,其中 Byte Buddy 构建了由类定义的方法的结构。它需要这样做,以解析层次结构中的桥接方法。这是一项相当昂贵的操作,但它不应该无休止地旋转。您是否有层次结构非常深(50 +级别)的课程?

相关内容

  • 没有找到相关文章

最新更新