Kubernetes Affinity阻止Jenkins worker在Main上运行

Question

Jenkins运行在EKS上，并且在Jenkins主pod和worker pod上都有关联规则。

这个想法是为了防止Jenkins工作pod运行在Jenkins主pod运行的同一个EKS工作节点上。

下面的规则可以工作，直到资源限制被推，这时Jenkins工作pod被调度到与Jenkins主pod相同的EKS工作节点上。

是否有亲和性/反亲和性规则来防止这种情况发生?

Jenkins的规则:

affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions: # assign to eks apps worker group
- key: node.app/group
operator: In
values:
- apps
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions: # don't assign to a node running jenkins main
- key: app.kubernetes.io/name
operator: In
values:
- jenkins
- key: app.kubernetes.io/component
operator: In
values:
- main
topologyKey: kubernetes.io/hostname
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions: # try not to assign to a node already running a jenkins worker
- key: app.kubernetes.io/name
operator: In
values:
- jenkins
- key: app.kubernetes.io/component
operator: In
values:
- worker
topologyKey: kubernetes.io/hostname

Jenkins worker的规则:

affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions: # assign to eks apps worker group
- key:  node.app/group
operator: In
values:
- apps
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions: # don't assign to a node running jenkins main
- key: app.kubernetes.io/name
operator: In
values:
- jenkins
- key: app.kubernetes.io/component
operator: In
values:
- main
topologyKey: kubernetes.io/hostname
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions: # try not to assign to a node already running a jenkins worker
- key: app.kubernetes.io/name
operator: In
values:
- jenkins
- key: app.kubernetes.io/component
operator: In
values:
- worker
topologyKey: kubernetes.io/hostname

Answer 1

这么低，你猜怎么着…主pod标签设置不正确。

现在你可以看到选择器标签显示在这里:

> aws-vault exec nonlive-build -- kubectl get po -n cicd --show-labels
NAME                                  READY   STATUS    RESTARTS   AGE    LABELS
jenkins-6597db4979-khxls              2/2     Running   0          4m8s   app.kubernetes.io/component=main,app.kubernetes.io/instance=jenkins

为了实现这一点，在values文件中添加了新的条目:

main:
metadata:
labels:
app.kubernetes.io/name: jenkins
app.kubernetes.io/component: main

和Helm _helpers。TPL模板相应更新:

{{- define "jenkins.selectorLabels" -}}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- if .Values.main.metadata.labels }}
{{- range $k, $v := .Values.main.metadata.labels }}
{{ $k }}: {{ $v }}
{{- end }}
{{- end }}
{{- end }}