AWS NLB+NGINX入口+websockets==502坏网关或坏握手

Question

入口控制器配置(不对下面的文件进行更改(

https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.41.2/deploy/static/provider/aws/deploy.yaml

服务

kind: Service
apiVersion: v1
metadata:
name: websockets-service
namespace: development
annotations:
argocd.argoproj.io/sync-wave: "10"
spec:
selector:
app: project-websocket
ports:
#- name: http
#  protocol: TCP
#  port: 80
#  targetPort: 9000
- name: https
protocol: TCP
port: 443
targetPort: 443
type: ClusterIP

入口

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: websockets-ingress
namespace: development
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
# nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/websocket-services: websockets-service
nginx.org/websocket-services: websockets-service
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header HOST $host;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass_request_headers on;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
argocd.argoproj.io/sync-wave: "10"
spec:
tls:
- hosts:
- backend-dev.project.com
secretName: tls-secret
rules:
- host: backend-dev.project.com
http:
paths:
- path: /ws/
backend:
serviceName: websockets-service
servicePort: 443

尝试各种CLI websocket客户端：

➜ ws --insecure wss://backend-dev.project.com/ws/table/123
websocket: bad handshake
➜ websocat -k  wss://backend-dev.project.com/ws/table/123
websocat: WebSocketError: Received unexpected status code (502 Bad Gateway)
websocat: error running

如果我从服务代理端口，一切都可以工作：

k port-forward svc/websockets-service 8443:443
Forwarding from 127.0.0.1:8443 -> 443
Forwarding from [::1]:8443 -> 443
Handling connection for 8443
➜ websocat -k wss://localhost:8443/ws/table/123
{"connect": {"channel_name": "specific.239f96e4a93a470688a12fc0bc8d0374!9d92256f6f65446386fb17b59d1aac57", "table_token": "123"}}
and in logs I see
127.0.0.1:50070 - - [21/Nov/2020:05:11:50] "WSCONNECTING /ws/table/123" - -
127.0.0.1:50070 - - [21/Nov/2020:05:11:50] "WSCONNECT /ws/table/123" - -

SSL信息

➜ curl -vvI https://backend-dev.project.com
*   Trying 44.235.41.143:443...
* Connected to backend-dev.project.com (44.235.41.143) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.project.com
*  start date: Oct 28 00:00:00 2020 GMT
*  expire date: Oct 28 23:59:59 2021 GMT
*  subjectAltName: host "backend-dev.project.com" matched cert's "*.project.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5579ce4a9af0)
> HEAD / HTTP/2
> Host: backend-dev.project.com
> user-agent: curl/7.69.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
HTTP/2 200 
< date: Sat, 21 Nov 2020 05:13:22 GMT
date: Sat, 21 Nov 2020 05:13:22 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-length: 143552
content-length: 143552
< vary: Accept-Encoding
vary: Accept-Encoding
< server-timing: TimerPanel_utime;dur=31.268000000011398;desc="User CPU time", TimerPanel_stime;dur=0.0;desc="System CPU time", TimerPanel_total;dur=31.268000000011398;desc="Total CPU time", TimerPanel_total_time;dur=31.340599060058594;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
server-timing: TimerPanel_utime;dur=31.268000000011398;desc="User CPU time", TimerPanel_stime;dur=0.0;desc="System CPU time", TimerPanel_total;dur=31.268000000011398;desc="Total CPU time", TimerPanel_total_time;dur=31.340599060058594;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-frame-options: DENY
x-frame-options: DENY
< vary: Cookie, Origin
vary: Cookie, Origin
< set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
set-cookie: csrftoken=zuBDnumLdRgcY5cICoIKnpAn98sdiGKdpiFKiiG7KvLr7bCPeOdMjzNK0mgTRUQZ; expires=Sat, 20 Nov 2021 05:13:22 GMT; Max-Age=31449600; Path=/; SameSite=Lax
< strict-transport-security: max-age=15724800; includeSubDomains
strict-transport-security: max-age=15724800; includeSubDomains
< 
* Connection #0 to host backend-dev.project.com left intact

Answer 1

Websockets和SSE(服务器发送的事件(是AWS的一大难题。对于SSE，我发现只有ALB才能正常工作。

好消息：AWS有ALB入口控制器
坏消息：就是这样。没有与nginx入口集成

您所能做的是混合使用alb-ingress控制器和nginx ingress控制器：
归根结底是安装alb-ingress-controller和ingress-nginx(裸机图(，然后创建一个从alb-ingress-controller指向ingress-nginx的NodePort服务的ingress资源。

所以类似于：

kind: Ingress
metadata:
name: "alb-ingress-to-nginx-ingress"
labels:
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: "internet-facing"
alb.ingress.kubernetes.io/target-type: "ip"
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:yyy:certificate/xxxx
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
spec:
rules:
- host: 'toto.edu'
http:
paths:
- backend:
serviceName: nginx-controller
servicePort: http

资源：中等物品github回购