我最初试图运行似乎被卡在crashbackoffloop中的作业。这是服务文件:
apiVersion: batch/v1
kind: Job
metadata:
name: es-setup-indexes
namespace: elk-test
spec:
template:
metadata:
name: es-setup-indexes
spec:
containers:
- name: es-setup-indexes
image: appropriate/curl
command: ['curl -H "Content-Type: application/json" -XPUT http://elasticsearch.elk-test.svc.cluster.local:9200/_template/filebeat -d@/etc/filebeat/filebeat.template.json']
volumeMounts:
- name: configmap-volume
mountPath: /etc/filebeat/filebeat.template.json
subPath: filebeat.template.json
restartPolicy: Never
volumes:
- name: configmap-volume
configMap:
name: elasticsearch-configmap-indexes
我尝试删除工作,但只有在运行以下命令时才有效:
kubectl delete job es-setup-indexes --cascade=false
之后,我在运行时注意到:
kubectl get pods -w
我会在错误状态下得到大量豆荚,我认为没有办法清理它们。这只是我运行时输出的一小部分样本:
es-setup-indexes-zvx9c 0/1 Error 0 20h
es-setup-indexes-zw23w 0/1 Error 0 15h
es-setup-indexes-zw57h 0/1 Error 0 21h
es-setup-indexes-zw6l9 0/1 Error 0 16h
es-setup-indexes-zw7fc 0/1 Error 0 22h
es-setup-indexes-zw9bw 0/1 Error 0 12h
es-setup-indexes-zw9ck 0/1 Error 0 1d
es-setup-indexes-zwf54 0/1 Error 0 18h
es-setup-indexes-zwlmg 0/1 Error 0 16h
es-setup-indexes-zwmsm 0/1 Error 0 21h
es-setup-indexes-zwp37 0/1 Error 0 22h
es-setup-indexes-zwzln 0/1 Error 0 22h
es-setup-indexes-zx4g3 0/1 Error 0 11h
es-setup-indexes-zx4hd 0/1 Error 0 21h
es-setup-indexes-zx512 0/1 Error 0 1d
es-setup-indexes-zx638 0/1 Error 0 17h
es-setup-indexes-zx64c 0/1 Error 0 21h
es-setup-indexes-zxczt 0/1 Error 0 15h
es-setup-indexes-zxdzf 0/1 Error 0 14h
es-setup-indexes-zxf56 0/1 Error 0 1d
es-setup-indexes-zxf9r 0/1 Error 0 16h
es-setup-indexes-zxg0m 0/1 Error 0 14h
es-setup-indexes-zxg71 0/1 Error 0 1d
es-setup-indexes-zxgwz 0/1 Error 0 19h
es-setup-indexes-zxkpm 0/1 Error 0 23h
es-setup-indexes-zxkvb 0/1 Error 0 15h
es-setup-indexes-zxpgg 0/1 Error 0 20h
es-setup-indexes-zxqh3 0/1 Error 0 1d
es-setup-indexes-zxr7f 0/1 Error 0 22h
es-setup-indexes-zxxbs 0/1 Error 0 13h
es-setup-indexes-zz7xr 0/1 Error 0 12h
es-setup-indexes-zzbjq 0/1 Error 0 13h
es-setup-indexes-zzc0z 0/1 Error 0 16h
es-setup-indexes-zzdb6 0/1 Error 0 1d
es-setup-indexes-zzjh2 0/1 Error 0 21h
es-setup-indexes-zzm77 0/1 Error 0 1d
es-setup-indexes-zzqt5 0/1 Error 0 12h
es-setup-indexes-zzr79 0/1 Error 0 16h
es-setup-indexes-zzsfx 0/1 Error 0 1d
es-setup-indexes-zzx1r 0/1 Error 0 21h
es-setup-indexes-zzx6j 0/1 Error 0 1d
kibana-kq51v 1/1 Running 0 10h
但是,如果我看着工作,我再也没有与之相关的:
$ kubectl get jobs --all-namespaces
NAMESPACE NAME DESIRED SUCCESSFUL AGE
kube-system configure-calico 1 1 46d
我还注意到kubectl的响应速度似乎很慢。我不知道豆荚是不断尝试重新启动还是处于某种破裂状态,但是如果有人可以让我知道如何进行故障排除,因为我没有在kubernetes遇到过这样的问题。
kube信息:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
kubectl delete pods --field-selector status.phase=Failed -n <your-namespace>
...清理您的名称空间中的任何失败的豆荚。
这是修复它的快速方法:(
kubectl get pods | grep Error | cut -d' ' -f 1 | xargs kubectl delete pod
编辑:如果您使用的是K8S的旧版本
-a
我有许多豆荚,这些豆荚陷入了以下状态
- containererCannotrun
- 错误
- imagepullbackoff
由于有效的原因,这些豆荚处于州的高度。但是即使在稍后解决问题之后,他们也没有自动清理。
清理,手动跟随事情无效:
# Doesn't work
kubectl get pods --field-selector status.phase=Error
# Doesn't work
kubectl get pods
--field-selector=status.phase=Error
# Doesn't work
kubectl get pods
--field-selector="status.phase=Error"
# Doesn't work
kubectl get pods
--field-selector="status.phase==Error"
遵循我们喜欢保留的状态pods滤波器的方法
# Validate list of pods.
# Please add more status that we don't want to delete
kubectl get pods
--field-selector="status.phase!=Succeeded,status.phase!=Running"
# Delete pods that matches the filter
kubectl delete pods
--field-selector="status.phase!=Succeeded,status.phase!=Running"
i通常使用此命令删除所有Error
POD。
kubectl delete pod `kubectl get pods --namespace <yournamespace> | awk '$3 == "Error" {print $1}'` --namespace <yournamespace>
解决方案如评论中提到的 @johnharris85。我必须手动删除所有豆荚。为此,我运行以下内容:
kubectl get pods -w | tee all-pods.txt
倾倒了我所有的豆荚,然后仅在我想要的东西上进行过滤和删除。
kubectl delete pod $(more all-pods.txt | grep es-setup-index | awk '{print $1}')
注意:我有大约9292个豆荚,大约需要1-2个小时才能删除它们。