GKE Nginx 入口控制器的入口未绑定到外部 IP



我正试图在GKE集群上添加一个NGINX Ingress控制器,现有的HAProxy Ingress控制器(在重写规则方面有一些问题(

首先,我尝试将控制器的服务公开为LoadBalancer类型。流量可以到达入口和后端,但它不适用于托管证书。

因此,我尝试使用L7负载均衡器(URL映射(将流量转发到GKE集群IP,并为我的入口控制器本身创建一个Ingress对象。

问题是,这个Ingress对象似乎没有绑定到外部IP。并且到域的路由产生";默认后端-404";回答

$ kubectl -n ingress-controller get service
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
haproxy-ingress           NodePort    172.16.xxx.xxx  <none>        80:31579/TCP,443:31769/TCP   595d
ingress-default-backend   ClusterIP   172.16.xxx.xxx  <none>        8080/TCP                     595d
nginx-ingress-svc         NodePort    172.16.xxx.xxx  <none>        80:32416/TCP,443:31299/TCP   2d17h
$ kubectl -n ingress-controller get ing
NAME                CLASS    HOSTS   ADDRESS          PORTS   AGE
haproxy-l7-ing      <none>   *       34.xxx.xxx.aaa   80      594d
ingress-nginx-ing   nginx    *       172.xxx.xxx.xxx  80      2d16h
$ gcloud compute addresses list --global --project my-project
NAME                    ADDRESS/RANGE   TYPE      PURPOSE  NETWORK  REGION  SUBNET  STATUS
my-ext-ip               34.xxx.xxx.aaa  EXTERNAL                                    IN_USE
my-test-ext-ip          34.xxx.xxx.bbb  EXTERNAL                                    IN_USE

在这种情况下,我假设ingress-nginx-ing应该绑定到34.xxx.xxx.bbb (my-test-ext-ip),就像haproxy-l7-ing绑定到34.xxx.xxx.aaa (my-ext-ip)一样,但它没有。

负载均衡器:

$ gcloud compute forwarding-rules list --global --project my-project
NAME                              REGION  IP_ADDRESS      IP_PROTOCOL  TARGET
haproxy-http-fwdrule                      34.xxx.xxx.aaa  TCP          haproxy-http-proxy
haproxy-https-fwdrule                     34.xxx.xxx.aaa  TCP          haproxy-https-proxy
nginx-http-fwdrule                        34.xxx.xxx.bbb  TCP          nginx-http-proxy
nginx-https-fwdrule                       34.xxx.xxx.bbb  TCP          nginx-https-proxy
$ gcloud compute target-http-proxies list --global --project my-project
NAME                URL_MAP
haproxy-http-proxy  haproxy-http-urlmap
nginx-http-proxy    nginx-https-urlmap
$ gcloud compute target-https-proxies list --global --project my-project
NAME                                  SSL_CERTIFICATES                    URL_MAP
haproxy-https-proxy                   default-cert,mcrt-xxxxxx-xxxxxx     haproxy-https-urlmap
nginx-https-proxy                     mcrt-xxxxxx-xxxxxx                  nginx-https-urlmap
$ gcloud compute url-maps list --global --project my-project
NAME                      DEFAULT_SERVICE
haproxy-https-urlmap      backendServices/k8s-be-xxxxxx--xxxxxx
haproxy-http-urlmap
nginx-https-urlmap        backendServices/nginx-lb-backendservice
$ gcloud compute backend-services list --global --project my-project
NAME                            BACKENDS                                         PROTOCOL
k8s-be-xxxxxx--xxxxxx           asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx  HTTP
nginx-lb-backendservice         asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx  HTTP

后端:asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx指向GKE集群。

K8S YAML是这样的:

---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
namespace: ingress-controller
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
kind: Service
apiVersion: v1
metadata:
name: nginx-ingress-svc
namespace: ingress-controller
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: NodePort
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
appProtocol: http
- name: https
port: 443
targetPort: https
protocol: TCP
appProtocol: https
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-nginx-ing
namespace: ingress-controller
labels:
app: ingress-nginx
tier: ingress
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
# kubernetes.io/ingress.allow-http: 'false'
kubernetes.io/ingress.global-static-ip-name: 'my-test-ext-ip'
ingress.kubernetes.io/url-map: nginx-https-urlmap
networking.gke.io/managed-certificates: 'my-managed-cert'
ingress.gcp.kubernetes.io/pre-shared-cert: 'default-cert'
spec:
ingressClassName: nginx
defaultBackend:
service:
name: nginx-ingress-svc
port:
number: 80

知道我在这里可能错过了什么吗?谢谢


更新

我为负载均衡器调整了一些配置,创建了自己的后端和健康检查,如下所示:

$ gcloud compute backend-services describe nginx-lb-backendservice --global
affinityCookieTtlSec: 0
backends:
- balancingMode: RATE
capacityScaler: 1.0
group: https://www.googleapis.com/compute/v1/projects/my-project/zones/asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx
maxRatePerInstance: 1.0
cdnPolicy:
cacheKeyPolicy:
includeHost: true
includeProtocol: true
includeQueryString: false
cacheMode: USE_ORIGIN_HEADERS
negativeCaching: false
requestCoalescing: true
serveWhileStale: 0
signedUrlCacheMaxAgeSec: '0'
connectionDraining:
drainingTimeoutSec: 0
creationTimestamp: '2022-01-07T00:48:38.900-08:00'
description: '{"kubernetes.io/service-name":"ingress-controller/nginx-ingress-svc","kubernetes.io/service-port":"80"}'
enableCDN: true
fingerprint: ****
healthChecks:
- https://www.googleapis.com/compute/v1/projects/mtb-development-project/global/healthChecks/nginx-lb-backend-healthcheck
id: '7699213954898870409'
kind: compute#backendService
loadBalancingScheme: EXTERNAL
logConfig:
enable: true
sampleRate: 1.0
name: nginx-lb-backendservice
port: 31579
portName: port31579
protocol: HTTP
selfLink: https://www.googleapis.com/compute/v1/projects/my-project/global/backendServices/nginx-lb-backendservice
sessionAffinity: NONE
timeoutSec: 30

然后,我将此注释添加到Ingressingress-nginx-ing:中

ingress.kubernetes.io/url-map: nginx-https-urlmap

后端状态为"健康",但不知何故ingress-nginx-ing仍然无法绑定到保留的外部IP。

此外,与HAProxy不同,它没有附加任何注释:ingress.kubernetes.io/backendsingress.kubernetes.io/https-forwarding-ruleingress.kubernetes.io/https-target-proxy

向myhost.mydomain/anything发送HTTP(S((解析为IP:34.xxx.xxx.bbb(仍然得到"0";默认后端-404";响应。

更新#2(有效!(

我在这里尝试了boredadel的答案,从ingress-nginx-ing中删除了ingressClassName: nginx,它似乎起了作用。

根据新的警告删除手动创建的LB对象,并调整自动生成的健康检查后,流量可以按预期到达API。

(混淆来源于同时具有kubernetes.io/ingress.class注释和实例中的ingressClassName。(

是的,所以我在你的YAML文件中看到的问题是,你试图使用NGINX IngressClass公开NGINX入口本身,这是行不通的。

您要做的是使用名为gce的GKE默认IngressClass公开NGINX。如果在Ingress对象中省略它,则为默认值。所以你的物体大致看起来像这个

HTTP LB(通过具有gce IngressClass的Ingress(->nginx服务->NGINX吊舱-->应用程序服务-->应用程序吊舱

我们这里有一个例子

然而,你需要记住的事情很少。NGINX Ingress控制器做的事情与GKE默认Ingress控制器几乎相同。它们都在你的应用程序前面设置了一个HTTP LoadBalancer。在这个设置中,您将尝试使用2个LoadBalancer,一个是通过Ingress提供的Google HTTP LB,另一个是NGINX。这意味着tcp终止次数是tcp终止次数的2倍,并可能导致延迟增加。需要牢记

托管证书只能与L7(HTTP(LoadBalancer一起使用,而不能与TCP一起使用。

我的理解是,你想在GKE上使用nginx作为Ingress控制器,但你想在L7 LoadBalancer后面公开它,这样你就可以使用谷歌托管证书了?

在我的例子中,它是在nginx-hhelm图表值中设置为false的publishservice。当然,您需要注释:kubernetes.io/ingress.class:nginx

最新更新