Kubernetes pod 自动重启,退出 137 代码



这个日志我从节点之一的 Kubernetes 退出容器中得到

可以取悦任何人 helo 我认为这是一个内存问题,但我为 pod 设置了足够的资源。

内存会随着时间的推移逐渐增加,因此内存泄漏可能会发生。请帮忙,谢谢。

它只是在完美地分期和重新开始生产。我也在想python-slim因为我在 docker 中使用的映像,所以内核或 Linux 本身会杀死我的 python 进程。

提前致谢

Nov 26 00:24:03 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: python3 invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null),  order=0, oom_score_adj=993
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  oom_kill_process+0x23e/0x490
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  out_of_memory+0x100/0x4c0
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  mem_cgroup_out_of_memory+0x3f/0x60
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  mem_cgroup_oom_synchronize+0x2dd/0x300
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  pagefault_out_of_memory+0x25/0x56
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 2106
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod2ef4b832-1101-11ea-9b9a-42010a8000a9: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod2ef4b832-1101-11ea-9b9a-42010a8000a9/4a728f33240d29d15761e3224c1c08a41943c233e8d2970b5068a19c95f1f3e1: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:48KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod2ef4b832-1101-11ea-9b9a-42010a8000a9/7dd87463773c32fbffad267b50f3986cdb969bd9915ab32cc371a50c9e2dc16f: cache:128KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod2ef4b832-1101-11ea-9b9a-42010a8000a9/6006c2b3ae7dcc7e6ddf41e765c747db71ed3b09c49e83cec281501ff848419e: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:132KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod2ef4b832-1101-11ea-9b9a-42010a8000a9/7f54c48546807d9430b82469e1968da2e83772b60c2c6b65a308d78b50eefc56: cache:0KB rss:1041756KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:1042088KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup out of memory: Kill process 3951244 (python3) score 2004 or sacrifice child
Nov 29 06:45:02 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: oom_reaper: reaped process 3951244 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Nov 29 06:45:24 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: python3 invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null),  order=0, oom_score_adj=993
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  oom_kill_process+0x23e/0x490
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  out_of_memory+0x100/0x4c0
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  mem_cgroup_out_of_memory+0x3f/0x60
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  mem_cgroup_oom_synchronize+0x2dd/0x300
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel:  pagefault_out_of_memory+0x25/0x56
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: memory: usage 1048576kB, limit 1048576kB, failcnt 795
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: memory+swap: usage 1048576kB, limit 9007199254740988kB, failcnt 0
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod06691003-1101-11ea-9b9a-42010a8000a9: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod06691003-1101-11ea-9b9a-42010a8000a9/c1265c7dc67ee140d0033c3527adcb4e47fded0e8ac27822701d2e56acbb528f: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:40KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod06691003-1101-11ea-9b9a-42010a8000a9/471908fde52e37475d1e454fd23755ac0066fd16f324aa1b8dcdae70ae3ee4db: cache:128KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod06691003-1101-11ea-9b9a-42010a8000a9/d29c34fa96c3350d5b5caf09f19be16d68d07bbd54dd80c7bb709f7d55937ae7: cache:0KB rss:44KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup stats for /kubepods/burstable/pod06691003-1101-11ea-9b9a-42010a8000a9/10686aeb2ec1f3054f2d6da37b75a74076c3c1ad61d0fda16601bcca8f66f8c2: cache:12KB rss:1042092KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:132KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:1042244KB inactive_file:0KB active_file:0KB unevictable:0KB
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: Memory cgroup out of memory: Kill process 3956354 (python3) score 2004 or sacrifice child
Nov 29 06:45:25 gke-cluster-highmem-pool-gen2-f2743e02-msv2 kernel: oom_reaper: reaped process 3956354 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

我从评论(社区维基(中发布了大卫的答案,因为它得到了OP的证实:

如果您看到该消息,则它是内核 OOM 杀手:您的节点是 内存不足。增加容器的资源请求以更接近 或等于资源限制可以有所帮助(通过保持 在同一节点上调度的其他进程(,但是如果您 有内存泄漏,你只需要修复它,这不是 真正可以从 Kubernetes 级别诊断出来的东西。

最新更新