我正试图在GKE集群上添加一个NGINX Ingress控制器,现有的HAProxy Ingress控制器(在重写规则方面有一些问题(
首先,我尝试将控制器的服务公开为LoadBalancer
类型。流量可以到达入口和后端,但它不适用于托管证书。
因此,我尝试使用L7负载均衡器(URL映射(将流量转发到GKE集群IP,并为我的入口控制器本身创建一个Ingress对象。
问题是,这个Ingress对象似乎没有绑定到外部IP。并且到域的路由产生";默认后端-404";回答
$ kubectl -n ingress-controller get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
haproxy-ingress NodePort 172.16.xxx.xxx <none> 80:31579/TCP,443:31769/TCP 595d
ingress-default-backend ClusterIP 172.16.xxx.xxx <none> 8080/TCP 595d
nginx-ingress-svc NodePort 172.16.xxx.xxx <none> 80:32416/TCP,443:31299/TCP 2d17h
$ kubectl -n ingress-controller get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
haproxy-l7-ing <none> * 34.xxx.xxx.aaa 80 594d
ingress-nginx-ing nginx * 172.xxx.xxx.xxx 80 2d16h
$ gcloud compute addresses list --global --project my-project
NAME ADDRESS/RANGE TYPE PURPOSE NETWORK REGION SUBNET STATUS
my-ext-ip 34.xxx.xxx.aaa EXTERNAL IN_USE
my-test-ext-ip 34.xxx.xxx.bbb EXTERNAL IN_USE
在这种情况下,我假设ingress-nginx-ing
应该绑定到34.xxx.xxx.bbb (my-test-ext-ip)
,就像haproxy-l7-ing
绑定到34.xxx.xxx.aaa (my-ext-ip)
一样,但它没有。
负载均衡器:
$ gcloud compute forwarding-rules list --global --project my-project
NAME REGION IP_ADDRESS IP_PROTOCOL TARGET
haproxy-http-fwdrule 34.xxx.xxx.aaa TCP haproxy-http-proxy
haproxy-https-fwdrule 34.xxx.xxx.aaa TCP haproxy-https-proxy
nginx-http-fwdrule 34.xxx.xxx.bbb TCP nginx-http-proxy
nginx-https-fwdrule 34.xxx.xxx.bbb TCP nginx-https-proxy
$ gcloud compute target-http-proxies list --global --project my-project
NAME URL_MAP
haproxy-http-proxy haproxy-http-urlmap
nginx-http-proxy nginx-https-urlmap
$ gcloud compute target-https-proxies list --global --project my-project
NAME SSL_CERTIFICATES URL_MAP
haproxy-https-proxy default-cert,mcrt-xxxxxx-xxxxxx haproxy-https-urlmap
nginx-https-proxy mcrt-xxxxxx-xxxxxx nginx-https-urlmap
$ gcloud compute url-maps list --global --project my-project
NAME DEFAULT_SERVICE
haproxy-https-urlmap backendServices/k8s-be-xxxxxx--xxxxxx
haproxy-http-urlmap
nginx-https-urlmap backendServices/nginx-lb-backendservice
$ gcloud compute backend-services list --global --project my-project
NAME BACKENDS PROTOCOL
k8s-be-xxxxxx--xxxxxx asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx HTTP
nginx-lb-backendservice asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx HTTP
后端:asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx
指向GKE集群。
K8S YAML是这样的:
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
name: nginx
namespace: ingress-controller
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
controller: k8s.io/ingress-nginx
---
kind: Service
apiVersion: v1
metadata:
name: nginx-ingress-svc
namespace: ingress-controller
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Local
type: NodePort
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
protocol: TCP
appProtocol: http
- name: https
port: 443
targetPort: https
protocol: TCP
appProtocol: https
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-nginx-ing
namespace: ingress-controller
labels:
app: ingress-nginx
tier: ingress
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
# kubernetes.io/ingress.allow-http: 'false'
kubernetes.io/ingress.global-static-ip-name: 'my-test-ext-ip'
ingress.kubernetes.io/url-map: nginx-https-urlmap
networking.gke.io/managed-certificates: 'my-managed-cert'
ingress.gcp.kubernetes.io/pre-shared-cert: 'default-cert'
spec:
ingressClassName: nginx
defaultBackend:
service:
name: nginx-ingress-svc
port:
number: 80
知道我在这里可能错过了什么吗?谢谢
更新
我为负载均衡器调整了一些配置,创建了自己的后端和健康检查,如下所示:
$ gcloud compute backend-services describe nginx-lb-backendservice --global
affinityCookieTtlSec: 0
backends:
- balancingMode: RATE
capacityScaler: 1.0
group: https://www.googleapis.com/compute/v1/projects/my-project/zones/asia-southeast1-a/instanceGroups/k8s-ig--xxxxxx
maxRatePerInstance: 1.0
cdnPolicy:
cacheKeyPolicy:
includeHost: true
includeProtocol: true
includeQueryString: false
cacheMode: USE_ORIGIN_HEADERS
negativeCaching: false
requestCoalescing: true
serveWhileStale: 0
signedUrlCacheMaxAgeSec: '0'
connectionDraining:
drainingTimeoutSec: 0
creationTimestamp: '2022-01-07T00:48:38.900-08:00'
description: '{"kubernetes.io/service-name":"ingress-controller/nginx-ingress-svc","kubernetes.io/service-port":"80"}'
enableCDN: true
fingerprint: ****
healthChecks:
- https://www.googleapis.com/compute/v1/projects/mtb-development-project/global/healthChecks/nginx-lb-backend-healthcheck
id: '7699213954898870409'
kind: compute#backendService
loadBalancingScheme: EXTERNAL
logConfig:
enable: true
sampleRate: 1.0
name: nginx-lb-backendservice
port: 31579
portName: port31579
protocol: HTTP
selfLink: https://www.googleapis.com/compute/v1/projects/my-project/global/backendServices/nginx-lb-backendservice
sessionAffinity: NONE
timeoutSec: 30
然后,我将此注释添加到Ingressingress-nginx-ing
:中
ingress.kubernetes.io/url-map: nginx-https-urlmap
后端状态为"健康",但不知何故ingress-nginx-ing
仍然无法绑定到保留的外部IP。
此外,与HAProxy不同,它没有附加任何注释:ingress.kubernetes.io/backends
、ingress.kubernetes.io/https-forwarding-rule
、ingress.kubernetes.io/https-target-proxy
。
向myhost.mydomain/anything发送HTTP(S((解析为IP:34.xxx.xxx.bbb
(仍然得到"0";默认后端-404";响应。
更新#2(有效!(
我在这里尝试了boredadel的答案,从ingress-nginx-ing
中删除了ingressClassName: nginx
,它似乎起了作用。
根据新的警告删除手动创建的LB对象,并调整自动生成的健康检查后,流量可以按预期到达API。
(混淆来源于同时具有kubernetes.io/ingress.class
注释和实例中的ingressClassName
。(
是的,所以我在你的YAML文件中看到的问题是,你试图使用NGINX IngressClass公开NGINX入口本身,这是行不通的。
您要做的是使用名为gce的GKE默认IngressClass公开NGINX。如果在Ingress对象中省略它,则为默认值。所以你的物体大致看起来像这个
HTTP LB(通过具有gce IngressClass的Ingress(->nginx服务->NGINX吊舱-->应用程序服务-->应用程序吊舱
我们这里有一个例子
然而,你需要记住的事情很少。NGINX Ingress控制器做的事情与GKE默认Ingress控制器几乎相同。它们都在你的应用程序前面设置了一个HTTP LoadBalancer。在这个设置中,您将尝试使用2个LoadBalancer,一个是通过Ingress提供的Google HTTP LB,另一个是NGINX。这意味着tcp终止次数是tcp终止次数的2倍,并可能导致延迟增加。需要牢记
托管证书只能与L7(HTTP(LoadBalancer一起使用,而不能与TCP一起使用。
我的理解是,你想在GKE上使用nginx作为Ingress控制器,但你想在L7 LoadBalancer后面公开它,这样你就可以使用谷歌托管证书了?
在我的例子中,它是在nginx-hhelm图表值中设置为false的publishservice。当然,您需要注释:kubernetes.io/ingress.class:nginx