小贝子编程

在AKS中使用管理身份的Azure密钥库



我正在Azure AKS中部署常规部署，我想使用keyvault来存储我的秘密以访问数据库。

这是我的部署文件:

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: sonarqube
name: sonarqube
spec:
selector:
matchLabels:
app: sonarqube
replicas: 1
template:
metadata:
labels:
app: sonarqube
spec:
containers:
- name: sonarqube
image: sonarqube:8.9-developer
resources:
requests:
cpu: 500m
memory: 1024Mi
limits:
cpu: 2000m
memory: 4096Mi
volumeMounts:
- mountPath: "/mnt/secrets/"
name: secrets-store-inline
- mountPath: "/opt/sonarqube/data/"
name: sonar-data-new
- mountPath: "/opt/sonarqube/extensions/plugins/"
name: sonar-extensions-new2
env:
- name: "SONARQUBE_JDBC_USERNAME"
valueFrom:
secretKeyRef:
name: test-secret
key: username
- name: "SONARQUBE_JDBC_PASSWORD"
valueFrom:
secretKeyRef:
name: test-secret
key: password
- name: "SONARQUBE_JDBC_URL"
valueFrom:
configMapKeyRef:
name: sonar-config
key: url
ports:
- containerPort: 9000
protocol: TCP
volumes:
- name: sonar-data-new
persistentVolumeClaim:
claimName: sonar-data-new
- name: sonar-extensions-new2
persistentVolumeClaim:
claimName: sonar-extensions-new2
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kv-provider"

这是我的秘密存储类:

kind: SecretProviderClass
metadata:
name: azure-kv-provider
spec:
provider: azure
secretObjects:
- data:
- key: username
objectName: username
- key: password
objectName: password
secretName: test-secret
type: Opaque
parameters:
usePodIdentity: "false"  
useAssignedIdentity: "true"
userAssignedIdentityID: "zzzz-zzzz-zzzz-zzzz-zzzz"
keyvaultName: "dbkvtz" 
cloudName: ""   
objects:  |
array:
- |
objectName: test
objectType: secret
objectAlias: username
objectVersion: "" 
- |
objectName: test
objectType: secret 
objectAlias: password  
objectVersion: ""       
resourceGroup: "myresourcegroup"  
subscriptionId: "yyyy-yyyy-yyyy-yyy-yyyy"       
tenantId: "xxxx-xxxx-xxxx-xxx-xxxx"

,"zzzz-zzzz-zzzz-zzzz-zzzz"为已创建的受管身份的Client ID。

在我创建的密钥库"dbkvtz"是通过"访问策略"添加的。我创建的受管身份。另一方面，在"管理身份"中;我无法在"Azure角色分配"中添加任何角色。没有为所选订阅找到角色分配。我不知道是否有必要在那里添加任何角色。

AKS集群是为系统分配托管身份而设置的。我想使用托管身份来访问密钥库，所以我创建了一个托管身份，客户端id为"zzzz-zzzz-zzzz-zzzz-zzzz";(哪里是"z">

我不太熟悉AKS中的密钥库集成，所以我不确定配置是否正确。

我得到这个错误:

kubectl description pods:

Normal   Scheduled    19m                  default-scheduler  Successfully assigned default/sonarqube-6bdb9cfc85-npbfw to aks-agentpool-16966606-vmss000000
Warning  FailedMount  5m43s (x5 over 16m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[secrets-store-inline sonar-data-new sonar-extensions-new2 default-token-t45tw]: timed out waiting for the condition
Warning  FailedMount  3m27s                kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[default-token-t45tw secrets-store-inline sonar-data-new sonar-extensions-new2]: timed out waiting for the condition
Warning  FailedMount  71s (x2 over 10m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[secrets-store-inline], unattached volumes=[sonar-data-new sonar-extensions-new2 default-token-t45tw secrets-store-inline]: timed out waiting for the condition
Warning  FailedMount  37s (x17 over 19m)   kubelet            MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/sonarqube-6bdb9cfc85-npbfw, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to create auth config, error: failed to get credentials, nodePublishSecretRef secret is not set

logs az aks show -g RG -n SonarQubeCluster

{
"aadProfile": null,
"addonProfiles": {
"azurepolicy": {
"config": null,
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
}
},
"httpApplicationRouting": {
"config": null,
"enabled": false,
"identity": null
},
"omsagent": {
"config": {
"logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
},
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
}
}
},
"agentPoolProfiles": [
{
"availabilityZones": [
"1"
],
"count": 2,
"enableAutoScaling": false,
"enableEncryptionAtHost": null,
"enableFips": false,
"enableNodePublicIp": null,
"enableUltraSsd": null,
"gpuInstanceProfile": null,
"kubeletConfig": null,
"kubeletDiskType": "OS",
"linuxOsConfig": null,
"maxCount": null,
"maxPods": 110,
"minCount": null,
"mode": "System",
"name": "agentpool",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
"nodeLabels": {},
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.7",
"osDiskSizeGb": 128,
"osDiskType": "Managed",
"osSku": "Ubuntu",
"osType": "Linux",
"podSubnetId": null,
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"proximityPlacementGroupId": null,
"scaleDownMode": null,
"scaleSetEvictionPolicy": null,
"scaleSetPriority": null,
"spotMaxPrice": null,
"tags": null,
"type": "VirtualMachineScaleSets",
"upgradeSettings": null,
"vmSize": "Standard_DS2_v2"
}
],
"apiServerAccessProfile": {
"authorizedIpRanges": null,
"enablePrivateCluster": false,
"enablePrivateClusterPublicFqdn": null,
"privateDnsZone": null
},
"autoScalerProfile": null,
"autoUpgradeProfile": null,
"azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
"disableLocalAccounts": null,
"diskEncryptionSetId": null,
"dnsPrefix": "SonarQubeCluster-dns",
"enablePodSecurityPolicy": null,
"enableRbac": true,
"extendedLocation": null,
"fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
"fqdnSubdomain": null,
"httpProxyConfig": null,
"id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
"identity": {
"principalId": "yy",
"tenantId": "rr",
"type": "SystemAssigned",
"userAssignedIdentities": null
},
"identityProfile": {
"kubeletidentity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
}
},
"kubernetesVersion": "1.20.7",
"linuxProfile": null,
"location": "southcentralus",
"maxAgentPools": 100,
"name": "SonarQubeCluster",
"networkProfile": {
"dnsServiceIp": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"loadBalancerProfile": {
"allocatedOutboundPorts": null,
"effectiveOutboundIPs": [
{
"id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
"resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
}
],
"idleTimeoutInMinutes": null,
"managedOutboundIPs": {
"count": 1
},
"outboundIPs": null,
"outboundIpPrefixes": null
},
"loadBalancerSku": "Standard",
"natGatewayProfile": null,
"networkMode": null,
"networkPlugin": "kubenet",
"networkPolicy": null,
"outboundType": "loadBalancer",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
},
"nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
"podIdentityProfile": null,
"powerState": {
"code": "Running"
},
"privateFqdn": null,
"privateLinkResources": null,
"provisioningState": "Succeeded",
"resourceGroup": "RG",
"securityProfile": null,
"servicePrincipalProfile": {
"clientId": "msi"
},
"sku": {
"name": "Basic",
"tier": "Free"
},
"type": "Microsoft.ContainerService/ManagedClusters",
"windowsProfile": null
}

你知道怎么了吗?

提前谢谢你。

SecretProviderClass中的userAssignedIdentityID必须是用户分配的Kubelet管理身份ID (NodePool的管理身份)，而不是为您的AKS创建的管理身份，因为卷将通过节点上的Kubelet访问。

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
name: azure-kvname-user-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: "<Kubelet identity ID>"
keyvaultName: "kvname"

你还需要为这个Kubelet Identity分配一个Role:

resource "azurerm_role_assignment" "akv_kubelet" {
scope                = azurerm_key_vault.akv.id
role_definition_name = "Key Vault Secrets Officer"
principal_id         = azurerm_kubernetes_cluster.aks.kubelet_identity[0].object_id
}

或

export KUBE_ID=$(az aks show -g <resource group> -n <aks cluster name> --query identityProfile.kubeletidentity.objectId -o tsv)
export AKV_ID=$(az keyvault show -g <resource group> -n <akv name> --query id -o tsv)
az role assignment create --assignee $KUBE_ID --role "Key Vault Secrets Officer" --scope $AKV_ID

用户分配标识的文档可以在这里找到，系统分配标识的文档可以在这里找到。

我想知道，如果当我创建一个新的AKS集群与选项"系统分配管理的身份"启用了新的"管理身份";是自动创建的吗?

我问这个是因为我没有使用任何其他"Managed Identity">而是我手工创建的那个

下面是下面的步骤:

  1. 新建"Managed Identity">

  2. In"Managed Identity">-&"访问控制(IAM)&";或"Azure角色分配";我没有权限添加任何角色，所以我把它作为默认值。

  3. 创建"密钥库"并添加几个"秘密"。

  4. In"Key Vault">-"访问策略">为"托管身份"添加新的访问策略;创建了代理池"SonarQubeCluster-agentpool">

当我检查"AKSclusterName"->"Properties">→点击"MC_xx_AKSclusterName_southcentralus">当我收到这条消息时，我似乎没有权限"您没有访问此资源的授权。">

如果它有助于理解一点问题，我附加了日志:

az aks show -g RG -n SonarQubeCluster 

{
"aadProfile": null,
"addonProfiles": {
"azurepolicy": {
"config": null,
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurepolicy-sonarqubecluster"
}
},
"httpApplicationRouting": {
"config": null,
"enabled": false,
"identity": null
},
"omsagent": {
"config": {
"logAnalyticsWorkspaceResourceID": "/subscriptions/xx/resourceGroups/DefaultResourceGroup-SCUS/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-44e26024-4977-4419-8d23-0e1e22e8804e-SCUS"
},
"enabled": true,
"identity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/omsagent-sonarqubecluster"
}
}
},
"agentPoolProfiles": [
{
"availabilityZones": [
"1"
],
"count": 2,
"enableAutoScaling": false,
"enableEncryptionAtHost": null,
"enableFips": false,
"enableNodePublicIp": null,
"enableUltraSsd": null,
"gpuInstanceProfile": null,
"kubeletConfig": null,
"kubeletDiskType": "OS",
"linuxOsConfig": null,
"maxCount": null,
"maxPods": 110,
"minCount": null,
"mode": "System",
"name": "agentpool",
"nodeImageVersion": "AKSUbuntu-1804gen2containerd-2021.07.25",
"nodeLabels": {},
"nodePublicIpPrefixId": null,
"nodeTaints": null,
"orchestratorVersion": "1.20.7",
"osDiskSizeGb": 128,
"osDiskType": "Managed",
"osSku": "Ubuntu",
"osType": "Linux",
"podSubnetId": null,
"powerState": {
"code": "Running"
},
"provisioningState": "Succeeded",
"proximityPlacementGroupId": null,
"scaleDownMode": null,
"scaleSetEvictionPolicy": null,
"scaleSetPriority": null,
"spotMaxPrice": null,
"tags": null,
"type": "VirtualMachineScaleSets",
"upgradeSettings": null,
"vmSize": "Standard_DS2_v2"
}
],
"apiServerAccessProfile": {
"authorizedIpRanges": null,
"enablePrivateCluster": false,
"enablePrivateClusterPublicFqdn": null,
"privateDnsZone": null
},
"autoScalerProfile": null,
"autoUpgradeProfile": null,
"azurePortalFqdn": "sonarqubecluster-dns-4b5e95d4.portal.hcp.southcentralus.azmk8s.io",
"disableLocalAccounts": null,
"diskEncryptionSetId": null,
"dnsPrefix": "SonarQubeCluster-dns",
"enablePodSecurityPolicy": null,
"enableRbac": true,
"extendedLocation": null,
"fqdn": "sonarqubecluster-dns-4b5e95d4.hcp.southcentralus.azmk8s.io",
"fqdnSubdomain": null,
"httpProxyConfig": null,
"id": "/subscriptions/xx/resourcegroups/RG/providers/Microsoft.ContainerService/managedClusters/SonarQubeCluster",
"identity": {
"principalId": "yy",
"tenantId": "rr",
"type": "SystemAssigned",
"userAssignedIdentities": null
},
"identityProfile": {
"kubeletidentity": {
"clientId": "yy",
"objectId": "zz",
"resourceId": "/subscriptions/xx/resourcegroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.ManagedIdentity/userAssignedIdentities/SonarQubeCluster-agentpool"
}
},
"kubernetesVersion": "1.20.7",
"linuxProfile": null,
"location": "southcentralus",
"maxAgentPools": 100,
"name": "SonarQubeCluster",
"networkProfile": {
"dnsServiceIp": "10.0.0.10",
"dockerBridgeCidr": "172.17.0.1/16",
"loadBalancerProfile": {
"allocatedOutboundPorts": null,
"effectiveOutboundIPs": [
{
"id": "/subscriptions/xx/resourceGroups/MC_xx_SonarQubeCluster_southcentralus/providers/Microsoft.Network/publicIPAddresses/nn",
"resourceGroup": "MC_xx_SonarQubeCluster_southcentralus"
}
],
"idleTimeoutInMinutes": null,
"managedOutboundIPs": {
"count": 1
},
"outboundIPs": null,
"outboundIpPrefixes": null
},
"loadBalancerSku": "Standard",
"natGatewayProfile": null,
"networkMode": null,
"networkPlugin": "kubenet",
"networkPolicy": null,
"outboundType": "loadBalancer",
"podCidr": "10.244.0.0/16",
"serviceCidr": "10.0.0.0/16"
},
"nodeResourceGroup": "MC_xx_SonarQubeCluster_southcentralus",
"podIdentityProfile": null,
"powerState": {
"code": "Running"
},
"privateFqdn": null,
"privateLinkResources": null,
"provisioningState": "Succeeded",
"resourceGroup": "RG",
"securityProfile": null,
"servicePrincipalProfile": {
"clientId": "msi"
},
"sku": {
"name": "Basic",
"tier": "Free"
},
"type": "Microsoft.ContainerService/ManagedClusters",
"windowsProfile": null
}

谢谢!

相关内容

  • 没有找到相关文章

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium