尝试使用 Azure Pod 标识失败多次，最终成功

Question

简介：我正在尝试让 Azure Pod 标识在我们的群集中工作，以从密钥保管库读取机密，并且大部分都成功了(到目前为止一切顺利)。目前，我们有两个密钥保管库，两个AzureIdentity，两个AzureIdentityBinding和两个Pod，每个密钥保管库都使用它们的密钥保管库。

在测试时，两个 Pod 是相等的 - 唯一的区别是它们的aadpodidbinding和指示要使用的密钥保管库的环境变量。启动时，Pod 连接到 KeyVault，读取两个值并使用Console.WriteLine打印它们。如果连接失败，Pod 将崩溃，k8s 将重新启动它。

问题：一个 Pod 可能会启动，能够立即从密钥保管库读取数据，而另一个 Pod 可能会崩溃并重新启动 - 似乎是 - 在能够获取访问令牌之前持续 5 次。

失败时，将引发以下异常：

Unhandled Exception: Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProviderException: Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/******************. Exception Message: Tried the following 3 methods to get an access token, but none of them worked.
Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/******************. Exception Message: Tried to get token using Managed Service Identity. Access token could not be acquired. MSI ResponseCode: Forbidden, Response: no AzureAssignedIdentity found for pod:default/kv-test-be
Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/******************. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Environment variable LOCALAPPDATA not set.
Parameters: Connection String: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/******************. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. No such file or directory
at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.GetAuthResultAsyncImpl(String authority, String resource, String scope)
at Microsoft.Azure.Services.AppAuthentication.AzureServiceTokenProvider.<get_KeyVaultTokenCallback>b__8_0(String authority, String resource, String scope)
at Microsoft.Azure.KeyVault.KeyVaultCredential.PostAuthenticate(HttpResponseMessage response)
at Microsoft.Azure.KeyVault.KeyVaultCredential.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.KeyVault.KeyVaultClient.GetSecretsWithHttpMessagesAsync(String vaultBaseUrl, Nullable`1 maxresults, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.KeyVault.KeyVaultClientExtensions.GetSecretsAsync(IKeyVaultClient operations, String vaultBaseUrl, Nullable`1 maxresults, CancellationToken cancellationToken)
at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.LoadAsync()
at Microsoft.Extensions.Configuration.AzureKeyVault.AzureKeyVaultConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at KeyvaultTest.Program.Main(String[] args) in /app/src/Program.cs:line 16

使用 FlexVolume 时的行为是相似的(最终我们的一组 pod 将在生产中使用)，但我发现使用两个相等的 pod 更容易与错误相关联。

在等待 pod 成功时，我在麦克风的日志中看到"绑定已删除"和"绑定已应用"消息。

我的问题：

• 这种行为是否"有意"，也许记录在某处？
• 是否有可以应用设置以使"删除 - 应用"周期更快？
• 还有什么可以做的来缩短 Pod 创建和应用标识绑定之间的时间吗？这个问题可能与 https://github.com/Azure/aad-pod-identity/issues/145 有关吗

源代码： 程序.cs

using System;
using System.IO;
using System.Threading;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
namespace KeyvaultTest
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Starting Keyvault read");
var configuration = new ConfigurationBuilder()
.AddAzureKeyVault()
.Build();
var test1 = configuration.GetValue<string>("jtest");
Console.WriteLine(test1);
var test2 = configuration.GetValue<string>("jtest:jtest");
Console.WriteLine(test2);
Console.WriteLine("Finished Keyvault read");
}
}
}

密钥保管库配置.cs.cs

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
using System.Threading;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Azure.KeyVault;
using Microsoft.Azure.Services.AppAuthentication;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Configuration.AzureKeyVault;
namespace KeyvaultTest
{
public static class KeyVaultConfiguration
{
public static IConfigurationBuilder AddAzureKeyVault(this IConfigurationBuilder builder)
{
var builtConfig = builder.Build();
var keyVaultName = Environment.GetEnvironmentVariable("KV_NAME");
if (string.IsNullOrWhiteSpace(keyVaultName))
{
throw new Exception("KV_NAME is not defined");
}
Console.WriteLine($"Using KV_NAME = {keyVaultName}");
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(
new KeyVaultClient.AuthenticationCallback(
azureServiceTokenProvider.KeyVaultTokenCallback));
builder.AddAzureKeyVault(
$"https://{keyVaultName}.vault.azure.net/",
keyVaultClient,
new DefaultKeyVaultSecretManager());
return builder;
}
}
}

任何帮助，提示或想法都非常感谢。

注意：我已经将同样的问题发布到项目github页面的问题板上 https://github.com/Azure/aad-pod-identity/issues/181

Answer 1

我们面临着同样的问题。我们通过升级 AAD 容器标识克服了此问题。我们的版本是 1.5，将这个升级到 1.7 解决了我们的问题。

在此之前，我们还将应用程序使用的软件包(Microsoft.Azure.Services.AppAuthentication & Azure.Security.KeyVault.Secrets)升级到最新版本，但这还不够。