WebSocket 连接到"wss://api.example.com/ws"失败：WebSocket 握手期间出错：意外响应代码：404

Question

我正在尝试在Google Kubernetes Engine和Istio中使用TLS设置websockets。

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: example-back-end
spec:
hosts:
- "api-dev.example.dev"
gateways:
- istio-system/example-gateway
http:
- match:
- uri:
prefix: /worker
route:
- destination:
host: worker
port:
number: 5001
- match:
- uri:
prefix: /
route:
- destination:
host: back-end
port:
number: 5000
- match:
- uri:
prefix: /ws
route:
- destination:
host: service-websocket
port:
number: 8080
websocketUpgrade: true

我已将 tls 证书和密钥挂载到我的 websocket 服务容器中。(我用于 api.example.com 的同一个(。

apiVersion: v1
kind: Service
metadata:
name: service-websocket
spec:
selector:
app: service-websocket
ports:
- port: 8080
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: service-websocket
spec:
selector:
matchLabels:
app: service-websocket
template:
metadata:
labels:
app: service-websocket
spec:
volumes:
- name: example-certificate
secret:
secretName: example-certificate
containers:
- name: service-websocket
image: gcr.io/example-project/service-websocket:latest
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8080
volumeMounts:
- name: example-certificate
mountPath: /var/secrets/tls

这是使用 ws 节点包的 websocket 服务器。

// web server
const https = require("https");
const config = require("./config");
const fs = require("fs");

const server = https.createServer({
cert: fs.readFileSync(config.TLS_CERT),
key: fs.readFileSync(config.TLS_KEY)
});
// websocket
const WebSocket = require("ws");
const url = require("url");
const wss = new WebSocket.Server({ noServer: true });
wss.on("connection", function connection(ws, req) {
const parameters = url.parse(req.url, true);

ws.on("message", function incoming(message) {
wss.clients.forEach(client => {
const msg = {
msg: "hello world from server" 
};
client.send(JSON.stringify(msg));
});
});
const msg = {
msg: "something"
};
ws.send(JSON.stringify(msg));
});
wss.on("error", () => console.log("error"));
server.on("upgrade", function upgrade(request, socket, head) {
const pathname = url.parse(request.url).pathname;
if (pathname === "/ws") {
wss.handleUpgrade(request, socket, head, function done(ws) {
wss.emit("connection", ws, request);
});
} else {
socket.destroy();
}
});
server.listen(8080);

从前端，我正在初始化 Websocket：

const ws = new WebSocket(`wss://api.example.com/ws`);

但是，我收到错误：

WebSocket connection to 'wss://api.example.com/ws' failed: Error during WebSocket handshake: Unexpected response code: 404

在我的 docker-compose 设置中，一切似乎都可以在本地正常工作。但是似乎无法弄清楚如何在 GKE + Istio 上完成这项工作。

更新 1/15/20

我更改了虚拟服务路由顺序。以前，/ws是在/之后。但是现在我从前端收到 503 错误。

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: example-back-end
spec:
hosts:
- "api-dev.example.dev"
gateways:
- istio-system/example-gateway
http:
- match:
- uri:
prefix: /ws
route:
- destination:
host: service-websocket
port:
number: 443
websocketUpgrade: true
- match:
- uri:
prefix: /worker
route:
- destination:
host: worker
port:
number: 5001
- match:
- uri:
prefix: /
route:
- destination:
host: back-end
port:
number: 5000

Answer 1

主要问题是TLS终止发生在网关而不是服务层。Istio 有一个 tls 直通模式，允许对 websocket 连接进行端到端加密。该解决方案涉及几个部分。

1. 创建 CNAME 记录以将ws.dev.example.dev指向dev.example.dev。这是关键，因为不支持 URI 前缀匹配，因此/ws以前不起作用。
2. 在网关上配置 TLS 直通，在虚拟服务上配置 SNI 主机。
3. 修改前端，以便在连接到 websocket 服务时使用ws.dev.example.dev。
4. 将 tls 证书和密钥挂载到 Websocket 服务中。(我们复制了一个存储为 Kubernetes 机密的现有证书(
5. 修改 Websocket 服务以使用该证书。

网关.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: example-gateway
namespace: istio-system
labels:
app: ingressgateway
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
protocol: HTTPS
name: https-ws
tls:
mode: PASSTHROUGH
hosts:
- "ws.dev.example.dev"

service-websocket-virtual-service.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: service-websocket
spec:
hosts:
- "ws.dev.example.dev"
gateways:
- istio-system/example-gateway
tls:
- match:
- port: 443
sni_hosts:
- "ws.dev.example.dev"
route:
- destination:
host: service-websocket
port:
number: 443

service-websocket.yaml

apiVersion: v1
kind: Service
metadata:
name: service-websocket
spec:
selector:
app: service-websocket
ports:
- port: 443
targetPort: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: service-websocket
spec:
selector:
matchLabels:
app: service-websocket
template:
metadata:
labels:
app: service-websocket
spec:
volumes:
- name: example-dev-certificate
secret:
secretName: example-dev-certificate
containers:
- name: service-websocket
image: gcr.io/example-dev/service-websocket:latest
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 443
volumeMounts:
- name: example-dev-certificate
mountPath: /var/secrets/tls

服务网络套接字节点服务器

const https = require("https");
const config = require("./config");
const fs = require("fs");
const server = https.createServer({
cert: fs.readFileSync(config.TLS_CERT),
key: fs.readFileSync(config.TLS_KEY)
});
// websocket
const WebSocket = require("ws");
const url = require("url");
const wss = new WebSocket.Server({ noServer: true });
wss.on("connection", function connection(ws, req) {
const parameters = url.parse(req.url, true);
ws.ideaRoom = { id: parseInt(parameters.query.ideaId) };
console.log("ws.ideaRoom", ws.ideaRoom);
ws.on("message", function incoming(message) {
console.log("received: %s", message);
console.log("wss.clients", wss.clients);
wss.clients.forEach(client => {
const msg = {
msg: "hello world from server " + ws.ideaRoom.id
};
client.send(JSON.stringify(msg));
});
});
const msg = {
msg: "something"
};
ws.send(JSON.stringify(msg));
});
wss.on("error", () => console.log("error"));
server.on("upgrade", function upgrade(request, socket, head) {
const pathname = url.parse(request.url).pathname;
console.log("pathname", pathname);
if (pathname === "/") {
wss.handleUpgrade(request, socket, head, function done(ws) {
wss.emit("connection", ws, request);
});
} else {
socket.destroy();
}
});
server.listen(443);

前端代码

const ws = new WebSocket(`wss://ws.dev.example.dev:443?ideaId=${idea.id}`);
ws.onopen = () => {
ws.send("hello world from client: " + idea.id);
};
ws.onerror = error => {
console.error(error);
};
ws.onmessage = e => {
console.log('e.data', e.data);
const idea_comment = JSON.parse(e.data);
console.log("idea_comment", idea_comment);
};