在Terraform上使用Kubernetes provider和私有GKE集群

Question

我想在Terraform上使用kubernetes提供程序与私有GKE集群进行交互。我可以成功地创建集群，但是我无法创建名称空间，我总是得到超时错误。身份验证不是问题，因为我可以在本地运行kubectl ...命令而没有问题。我认为这个问题与集群是私有的这一事实有关(因为我发现的所有示例都与公共集群有关)。有人知道如何将kubernetes提供程序连接到私有GKE集群吗?

我的main.tf文件:

provider "google" {
project = "<PROJECT_ID>"
}
variable "cluster_name" {
default = "<CLUSTER_NAME>"
}
resource "google_container_cluster" "composer_cluster" {
name      = var.cluster_name
location  = "europe-west1-b"
# Node
initial_node_count = 1
node_config {
disk_size_gb  = 100
disk_type     = "pd-standard"
machine_type  = "n1-standard-4"
metadata      = {
disable-legacy-endpoints= true
}
oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
service_account = "<SERVICE_ACCOUNT>"
}
# Network
network     = "<NETWORK>"
subnetwork  = "<SUBNETWORK>"
# IP allocation
private_cluster_config {
enable_private_endpoint= true
enable_private_nodes= true
master_global_access_config {
enabled= true
}
master_ipv4_cidr_block= "172.16.32.0/28"
}
ip_allocation_policy {
cluster_ipv4_cidr_block= "10.92.0.0/14"
services_ipv4_cidr_block= "10.82.240.0/20"
}
# Security
enable_kubernetes_alpha= false
enable_legacy_abac= false
enable_intranode_visibility= true
master_authorized_networks_config {}
network_policy {
enabled= true
provider= "CALICO"
}
enable_shielded_nodes= true
# Timeouts
timeouts {
create = "30m"
update = "40m"
}
}
data "google_client_config" "current" {}
provider "kubernetes" {
host    = google_container_cluster.composer_cluster.private_cluster_config[0].private_endpoint
token   = data.google_client_config.current.access_token
client_certificate = base64decode(google_container_cluster.composer_cluster.master_auth[0].client_certificate)
client_key = base64decode(google_container_cluster.composer_cluster.master_auth[0].client_key)
cluster_ca_certificate = base64decode(google_container_cluster.composer_cluster.master_auth[0].cluster_ca_certificate)
}
resource "null_resource" "get-credentials" {
depends_on = [google_container_cluster.composer_cluster] 
provisioner "local-exec" {
command = "gcloud container clusters get-credentials ${google_container_cluster.composer_cluster.name} --internal-ip --zone europe-west1-b --project <PROJECT_ID>"
}
}
resource "kubernetes_namespace" "namespace" {
metadata {
labels = {
app = "create-namespace"
}
name = "<NAMESPACE>"
}
depends_on = [null_resource.get-credentials]
}

输出:

oogle_container_cluster.composer_cluster: Creating...
google_container_cluster.composer_cluster: Still creating... [10s elapsed]
google_container_cluster.composer_cluster: Still creating... [20s elapsed]
google_container_cluster.composer_cluster: Still creating... [30s elapsed]
google_container_cluster.composer_cluster: Still creating... [40s elapsed]
google_container_cluster.composer_cluster: Still creating... [50s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [1m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [2m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [3m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m0s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m10s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m20s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m30s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m40s elapsed]
google_container_cluster.composer_cluster: Still creating... [4m50s elapsed]
google_container_cluster.composer_cluster: Still creating... [5m0s elapsed]
google_container_cluster.composer_cluster: Creation complete after 5m2s [id=projects/<PROJECT_ID>/locations/europe-west1-b/clusters/<CLUSTER_NAME>]
kubernetes_namespace.namespace: Creating...
kubernetes_namespace.namespace: Still creating... [10s elapsed]
kubernetes_namespace.namespace: Still creating... [20s elapsed]
kubernetes_namespace.namespace: Still creating... [30s elapsed]
Error: Post "https://172.16.32.2/api/v1/namespaces": dial tcp 172.16.32.2:443: i/o timeout

Answer 1

似乎仍然是一个身份验证问题，我已经成功地运行gcloud模块对GKE集群进行身份验证:

module "gcloud" {
source  = "terraform-google-modules/gcloud/google"
version = "~> 0.5"
platform = "linux"
create_cmd_entrypoint  = "gcloud"
create_cmd_body        = "container clusters get-credentials ${google_container_cluster.composer_cluster.name} --region=${var.zone}"
}
provider "kubernetes" {
# the authorization is handled by running gcloud clusters get-credentials using the gcloud terraform module
}
resource "kubernetes_deployment" "main" {
....
}