小贝子编程

GKE网关API httproute在负载均衡器和应用程序之间的https不工作



我正在测试网关API与GKE(版本1.21.11-gke.1100)。我使用gatewayClassName: gke-l7-rilb作为网关，在客户端和网关之间使用TLS。HTTPS使用托管的区域SSL证书在客户端和负载均衡器之间完美地工作。

我有2个http路由引用2个kube服务(backendRefs)。一个服务可以通过HTTP访问，另一个可以通过HTTPS访问(Argo工作流项目中的Argo服务器服务，如果它可能有帮助的话)。

当我使用HTTP创建引用服务的httproute时，GCP负载平衡器后端服务被创建并正常工作，没有任何问题(健康)。

但是当我创建引用argo服务的httproute时，创建了一个GCP负载均衡器后端服务，但不工作(不健康)，端点协议设置为HTTP而不是HTTPS。您应该知道，我确保在argo-server服务中添加注释cloud.google.com/app-protocols: '{"web":"HTTPS"}'，以启用负载均衡器和argo-server应用程序之间的HTTPS。

如果我使用入口资源和相同的argo服务定义创建相同的网关api配置，则(GCP负载均衡器后端服务的)端点协议被正确设置为HTTPS，并且完全健康且工作。

就像网关API的httproute一样，GKE网关控制器没有考虑到cloud.google.com/app-protocols服务注释，尽管这里提到它与网关API相关。

编辑1:添加yaml文件

网关:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"Gateway","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"regional-internal-https","namespace":"exposition"},"spec":{"addresses":[{"type":"NamedAddress","value":"dev-gateway-internal-lb-static-ip"}],"gatewayClassName":"gke-l7-rilb","listeners":[{"allowedRoutes":{"kinds":[{"kind":"HTTPRoute"}],"namespaces":{"from":"Selector","selector":{"matchLabels":{"exposed":"true"}}}},"name":"https","port":443,"protocol":"HTTPS","tls":{"mode":"Terminate","options":{"networking.gke.io/pre-shared-certs":"plat-dev-europe-west1"}}}]}}
networking.gke.io/addresses: ""
networking.gke.io/backend-services: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
networking.gke.io/firewalls: ""
networking.gke.io/forwarding-rules: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
networking.gke.io/health-checks: gkegw1-bkib-argo-argo-server-2746-8ktcvo8d0ktp,
gkegw1-bkib-demo-application-demo-service-80-y5bgcnm71kjv, gkegw1-bkib-exposition-gw-serve404-80-pciznuyt569p
networking.gke.io/last-reconcile-time: "2022-06-16T15:57:45Z"
networking.gke.io/ssl-certificates: ""
networking.gke.io/target-proxies: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
networking.gke.io/url-maps: gkegw1-bkib-exposition-regional-internal-https-tqsh4njw7io8
creationTimestamp: "2022-06-15T08:28:20Z"
finalizers:
- gateway.finalizer.networking.gke.io
generation: 1
labels:
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
managedFields:
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
.: {}
f:addresses: {}
f:gatewayClassName: {}
f:listeners:
.: {}
k:{"name":"https"}:
.: {}
f:allowedRoutes:
.: {}
f:kinds: {}
f:namespaces:
.: {}
f:from: {}
f:selector:
.: {}
f:matchLabels:
.: {}
f:exposed: {}
f:name: {}
f:port: {}
f:protocol: {}
f:tls:
.: {}
f:mode: {}
f:options:
.: {}
f:networking.gke.io/pre-shared-certs: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-06-15T08:28:20Z"
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:networking.gke.io/addresses: {}
f:networking.gke.io/backend-services: {}
f:networking.gke.io/firewalls: {}
f:networking.gke.io/forwarding-rules: {}
f:networking.gke.io/health-checks: {}
f:networking.gke.io/last-reconcile-time: {}
f:networking.gke.io/ssl-certificates: {}
f:networking.gke.io/target-proxies: {}
f:networking.gke.io/url-maps: {}
f:finalizers:
.: {}
v:"gateway.finalizer.networking.gke.io": {}
f:status:
f:addresses: {}
manager: GoogleGKEGatewayController
operation: Update
time: "2022-06-15T08:30:16Z"
name: regional-internal-https
namespace: exposition
resourceVersion: "42337844"
uid: 59333aea-1a79-4e9b-afbc-595ae9ccdfd7
spec:
addresses:
- type: NamedAddress
value: dev-gateway-internal-lb-static-ip
gatewayClassName: gke-l7-rilb
listeners:
- allowedRoutes:
kinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespaces:
from: Selector
selector:
matchLabels:
exposed: "true"
name: https
port: 443
protocol: HTTPS
tls:
mode: Terminate
options:
networking.gke.io/pre-shared-certs: plat-dev-europe-west1
status:
addresses:
- type: IPAddress
value: 10.163.112.28
conditions:
- lastTransitionTime: "1970-01-01T00:00:00Z"
message: Waiting for controller
reason: NotReconciled
status: Unknown
type: Scheduled
  • Httproute:
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1alpha2","kind":"HTTPRoute","metadata":{"annotations":{},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"hostnames":["argo-server.plat.dev.df.gcp.corp.modified.com"],"parentRefs":[{"kind":"Gateway","name":"regional-internal-https","namespace":"exposition"}],"rules":[{"backendRefs":[{"name":"argo-server","port":2746}]}]}}
creationTimestamp: "2022-06-15T12:27:04Z"
generation: 1
labels:
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
managedFields:
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
.: {}
f:hostnames: {}
f:parentRefs: {}
f:rules: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-06-15T12:27:04Z"
- apiVersion: gateway.networking.k8s.io/v1alpha2
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:parents: {}
manager: GoogleGKEGatewayController
operation: Update
time: "2022-06-15T12:29:02Z"
name: argo-server
namespace: argo
resourceVersion: "42362026"
uid: 981ce997-c574-4878-bec1-b03c7707838c
spec:
hostnames:
- argo-server.plat.dev.df.gcp.corp.modified.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: regional-internal-https
namespace: exposition
rules:
- backendRefs:
- group: ""
kind: Service
name: argo-server
port: 2746
weight: 1
matches:
- path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2022-06-16T17:00:11Z"
message: ""
reason: RouteAccepted
status: "True"
type: Accepted
- lastTransitionTime: "2022-06-16T17:00:11Z"
message: ""
reason: ReconciliationSucceeded
status: "True"
type: Reconciled
controllerName: networking.gke.io/gateway
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: regional-internal-https
namespace: exposition
  • 服务:
apiVersion: v1
kind: Service
metadata:
annotations:
cloud.google.com/app-protocols: '{"web":"HTTPS"}'
cloud.google.com/backend-config: '{"default": "argo-server-backendconfig"}'
cloud.google.com/neg: '{"exposed_ports":{"2746":{}}}'
cloud.google.com/neg-status: '{"network_endpoint_groups":{"2746":"k8s1-f83345f9-argo-argo-server-2746-4d39c835"},"zones":["europe-west1-c"]}'
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/app-protocols":"{"web":"HTTPS"}","cloud.google.com/backend-config":"{"default": "argo-server-backendconfig"}","cloud.google.com/neg":"{"ingress": true}","cluster-autoscaler.kubernetes.io/safe-to-evict":"true"},"labels":{"app.kubernetes.io/managed-by":"gcp-cloud-build-deploy"},"name":"argo-server","namespace":"argo"},"spec":{"ports":[{"name":"web","port":2746,"targetPort":2746}],"selector":{"app":"argo-server"}}}
creationTimestamp: "2022-06-15T11:44:07Z"
labels:
app.kubernetes.io/managed-by: gcp-cloud-build-deploy
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:cloud.google.com/app-protocols: {}
f:cloud.google.com/backend-config: {}
f:cluster-autoscaler.kubernetes.io/safe-to-evict: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
f:ports:
.: {}
k:{"port":2746,"protocol":"TCP"}:
.: {}
f:name: {}
f:port: {}
f:protocol: {}
f:targetPort: {}
f:selector:
.: {}
f:app: {}
f:sessionAffinity: {}
f:type: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-06-15T12:27:23Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cloud.google.com/neg: {}
manager: GoogleGKEGatewayController
operation: Update
time: "2022-06-15T12:28:06Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:cloud.google.com/neg-status: {}
manager: glbc
operation: Update
time: "2022-06-15T12:28:06Z"
name: argo-server
namespace: argo
resourceVersion: "41692832"
uid: 25024d53-1d31-4165-8033-1843ec5d72ec
spec:
clusterIP: 10.163.247.121
clusterIPs:
- 10.163.247.121
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: web
port: 2746
protocol: TCP
targetPort: 2746
selector:
app: argo-server
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}

我找到了一个解决方案，我认为这是一个变通的办法。

  1. 使用networking.gke.io/app-protocols: '{"web":"HTTPS"}'注释而不是cloud.google.com/app-protocols: '{"web":"HTTPS"}'。该注释将在服务级别使用，其中web是端口的名称。这将启用负载均衡器和应用程序之间的HTTPS(为指定的HTTPRoute创建的后端服务的端点协议)。这是与gatewayClassName: gke-l7-rilb一个区域内部负载均衡器完美工作。

  2. 使用cloud.google.com/v1 BackendConfig创建自定义健康检查，其中将类型设置为HTTPS，端口设置为2746。这里有更多的细节https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#direct_health对于ingress, GCE入口控制器从应用程序就绪探测器自动创建此健康检查，但显然此功能尚未在GKE网关控制器中实现。

  3. 确保您有防火墙规则允许在2746端口上进行Google Cloud健康检查的入口流量使用入口，GCE入口控制器自动创建所需的防火墙规则，但显然此功能尚未在GKE网关控制器中实现。

最后我说这是一个变通方法，因为我想象并希望GKE网关控制器的未来版本将解决我上面提到的3个问题或要点。

相关内容

  • 没有找到相关文章

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium