Kubernetes cronjob和更新一个秘密

Question

下面是我的python脚本来更新一个秘密，这样我就可以使用kubectl部署到kubernetes。 所以它工作正常。 但是我想创建一个 kubernetes cron 作业，它将运行一个 docker 容器来更新 kubernetes 集群中的密钥。 我该怎么做？ aws 秘密仅持续 12 小时，我必须从集群内重新生成，以便在 pod 崩溃等情况下我可以拉动......

这是我在 kubernetes 中可以访问的内部 API？

cmd = """aws ecr get-login --no-include-email --region us-east-1 > aws_token.txt"""
run_bash(cmd)

f = open('aws_token.txt').readlines()
TOKEN = f[0].split(' ')[5]

SECRET_NAME = "%s-ecr-registry" % (self.region)

cmd = """kubectl delete secret --ignore-not-found %s -n %s""" % (SECRET_NAME,namespace)
print (cmd)
run_bash(cmd)
cmd = """kubectl create secret docker-registry %s --docker-server=https://%s.dkr.ecr.%s.amazonaws.com --docker-username=AWS --docker-password="%s" --docker-email="[e-mail]" -n %s """ % (SECRET_NAME,self.aws_account_id,self.region,TOKEN,namespace)
print (cmd)
run_bash(cmd)
cmd = "kubectl describe secrets/%s-ecr-registry -n %s" % (self.region,namespace)
print (cmd)
run_bash(cmd)
cmd = "kubectl get secret %s-ecr-registry -o yaml -n %s" % (self.region,namespace)
print (cmd)

Answer 1

碰巧的是，我真的完成了这项工作。

以下是设置 cronjob 以滚动您的 AWS docker 登录令牌，然后每 6 小时重新登录 ECR 所需的一切。只需将 {{ 变量 }} 替换为您自己的实际值即可。

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: {{ namespace }}
  name: ecr-cred-helper
rules:
- apiGroups: [""]
  resources:
  - secrets
  - serviceaccounts
  - serviceaccounts/token
  verbs:
  - 'delete'
  - 'create'
  - 'patch'
  - 'get'
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: ecr-cred-helper
  namespace: {{ namespace }}
subjects:
- kind: ServiceAccount
  name: sa-ecr-cred-helper
  namespace: {{ namespace }}
roleRef:
  kind: Role
  name: ecr-cred-helper
  apiGroup: ""
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-ecr-cred-helper
  namespace: {{ namespace }}
---
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  annotations:
  name: ecr-cred-helper
  namespace: {{ namespace }}
spec:
  concurrencyPolicy: Allow
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata:
      creationTimestamp: null
    spec:
      template:
        metadata:
          creationTimestamp: null
        spec:
          serviceAccountName: sa-ecr-cred-helper
          containers:
          - command:
            - /bin/sh
            - -c
            - |-
              TOKEN=`aws ecr get-login --region ${REGION} --registry-ids ${ACCOUNT} | cut -d' ' -f6`
              echo "ENV variables setup done."
              kubectl delete secret -n {{ namespace }} --ignore-not-found $SECRET_NAME
              kubectl create secret -n {{ namespace }} docker-registry $SECRET_NAME 
              --docker-server=https://{{ ECR_REPOSITORY_URL }} 
              --docker-username=AWS 
              --docker-password="${TOKEN}" 
              --docker-email="${EMAIL}"
              echo "Secret created by name. $SECRET_NAME"
              kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"'$SECRET_NAME'"}]}' -n {{ namespace }}
              echo "All done."
            env:
            - name: AWS_DEFAULT_REGION
              value: eu-west-1
            - name: AWS_SECRET_ACCESS_KEY
              value: '{{ AWS_SECRET_ACCESS_KEY }}'
            - name: AWS_ACCESS_KEY_ID
              value: '{{ AWS_ACCESS_KEY_ID }}'
            - name: ACCOUNT
              value: '{{ AWS_ACCOUNT_ID }}'
            - name: SECRET_NAME
              value: '{{ imagePullSecret }}'
            - name: REGION
              value: 'eu-west-1'
            - name: EMAIL
              value: '{{ ANY_EMAIL }}'
            image: odaniait/aws-kubectl:latest
            imagePullPolicy: IfNotPresent
            name: ecr-cred-helper
            resources: {}
            securityContext:
              capabilities: {}
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
          dnsPolicy: Default
          hostNetwork: true
          restartPolicy: Never
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 30
  schedule: 0 */6 * * *
  successfulJobsHistoryLimit: 3
  suspend: false

Answer 2

我添加了使用 cronjob 在命名空间之间复制机密的解决方案，因为这是在使用 CronJob 搜索机密复制时给我的堆栈溢出答案

在源命名空间中，您需要定义Role、RoleBinding 和"服务帐户"

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-user-user-secret-service-account
  namespace: source-namespace 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: demo-user-role
  namespace: source-namespace 
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    # Secrets you want to have access in your namespace
    resourceNames: ["demo-user" ]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: demo-user-cron-role-binding
  namespace: source-namespace 
subjects:
- kind: ServiceAccount
  name: demo-user-user-secret-service-account
  namespace: source-namespace 
roleRef:
  kind: Role
  name: demo-user-role
  apiGroup: ""

CronJob定义将如下所示：

apiVersion: batch/v1
kind: CronJob
metadata:
  name: demo-user-user-secret-copy-cronjob 
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 5
  successfulJobsHistoryLimit: 3
  startingDeadlineSeconds: 10
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: demo-user-user-secret-copy-cronjob
            image: bitnami/kubectl:1.25.4-debian-11-r6
            imagePullPolicy: IfNotPresent
            command:
              - "/bin/bash"
              - "-c"
              - "kubectl -n source-namespace get secret demo-user -o json | 
                jq 'del(.metadata.creationTimestamp, .metadata.uid, .metadata.resourceVersion, .metadata.ownerReferences, .metadata.namespace)' > /tmp/demo-user-secret.json && 
                kubectl apply --namespace target-namespace -f /tmp/demo-user-secret.json"
          restartPolicy: Never 
          securityContext:
            privileged: false
            allowPrivilegeEscalation: true
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            capabilities:
              drop: [ "all" ]
          serviceAccountName: demo-user-user-secret-service-account

在目标命名空间中，还需要Role和RoleBinding，以便源命名空间中的CronJob可以复制机密。

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: target-namespace 
  name: demo-user-role
rules:
- apiGroups: [""]
  resources:
  - secrets
  verbs:
  - 'list'
  - 'delete'
  - 'create'
  - 'patch'
  - 'get'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: demo-user-role-binding
  namespace: target-namespace 
subjects:
- kind: ServiceAccount
  name: demo-user-user-secret-service-account
  namespace: source-namespace 
roleRef:
  kind: Role
  name: demo-user-role
  apiGroup: ""

在目标命名空间部署中，可以将机密作为常规文件读取。

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  replicas: 1
    ...
    spec:
      containers:
      - name: my-app
        image: [image-name]
        volumeMounts:
          - name: your-secret
            mountPath: /opt/your-secret
            readOnly: true
      volumes:
        - name: your-secret
          secret:
            secretName: demo-user
            items:
              - key: ca.crt
                path: ca.crt
              - key: user.crt
                path: user.crt
              - key: user.key
                path: user.key
              - key: user.p12
                path: user.p12
              - key: user.password
                path: user.password