Kubernetes Haproxy Ingress+Nginx作为后端Https

Question

我使用这个软

Kubernetes 1.11.5
Haproxy：最后一个
Nginx:1.15.7

我对我购买的由comodo CA 制造的证书保密

得到这个错误：

Error code: SSL_ERROR_RX_RECORD_TOO_LONG

这是我的配置Haproxy入口

---
apiVersion: v1
kind: Namespace
metadata:
name: ingress-controller
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ingress-controller
namespace: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: ingress-controller
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: ingress-controller
namespace: ingress-controller
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- create
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-controller
subjects:
- kind: ServiceAccount
name: ingress-controller
namespace: ingress-controller
- apiGroup: rbac.authorization.k8s.io
kind: User
name: ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: ingress-controller
namespace: ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-controller
subjects:
- kind: ServiceAccount
name: ingress-controller
namespace: ingress-controller
- apiGroup: rbac.authorization.k8s.io
kind: User
name: ingress-controller
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
run: ingress-default-backend
name: ingress-default-backend
namespace: ingress-controller
spec:
selector:
matchLabels:
run: ingress-default-backend
template:
metadata:
labels:
run: ingress-default-backend
spec:
containers:
- name: ingress-default-backend
image: gcr.io/google_containers/defaultbackend:1.0
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
name: ingress-default-backend
namespace: ingress-controller
spec:
ports:
- port: 8080
selector:
run: ingress-default-backend
---
apiVersion: v1
kind: ConfigMap
metadata:
name: haproxy-ingress
namespace: ingress-controller
data:
ssl-options: force-tlsv12
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
run: haproxy-ingress
name: haproxy-ingress
namespace: ingress-controller
spec:
updateStrategy:
type: RollingUpdate
selector:
matchLabels:
run: haproxy-ingress
template:
metadata:
labels:
run: haproxy-ingress
spec:
hostNetwork: true
nodeSelector:
role: edge-router
serviceAccountName: ingress-controller
containers:
- name: haproxy-ingress
image: quay.io/jcmoraisjr/haproxy-ingress
args:
- --default-backend-service=$(POD_NAMESPACE)/ingress-default-backend
- --default-ssl-certificate=default/tls-secret
- --configmap=$(POD_NAMESPACE)/haproxy-ingress
- --sort-backends
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
- name: stat
containerPort: 1936
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace

这是我的应用程序和它的入口配置

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: meteo
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: meteo
template:
metadata:
labels:
app: meteo
spec:
containers:
- name: meteo
image: devprofi/meteo:v39
ports:
- containerPort: 443
imagePullSecrets:
- name: meteo-secret
---
apiVersion: v1
kind: Service
metadata:
name: meteo-svc
namespace: default
spec:
type: NodePort
ports:
#    - port: 80
#      targetPort: 80
#      protocol: TCP
#      name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
selector:
app: meteo
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/ssl-passthrough: "true"
kubernetes.io/ingress.class: "haproxy"
ingress.kubernetes.io/secure-backends: "true"   
ingress.kubernetes.io/backend-protocol: "HTTPS"   
name: meteo-ingress
namespace: default
spec:
tls:
- hosts:
- meteotravel.ru
secretName: cafe-secret  # this is another copy of secret made from my buyed cert and key
rules:
- host: meteotravel.ru
http:
paths:
- path: /
backend:
serviceName: meteo-svc
servicePort: 443

我尝试这个命令，得到错误

openssl s_client -connect meteotravel.ru:443
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL/CN=meteotravel.ru
issuer=/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 6055 bytes and written 312 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol  : TLSv1.2
Cipher    : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: BDD996AF8814404E3E385A6FBE49F56CA2668C54FD157FD3FB28F38DB64F771E
Session-ID-ctx: 
Master-Key: F8EB4A4DA674F286E44C71605DF1D7DE4A6FE58D249162B086CE17E899FAC88CFA213018F89B8A9939CB842639D2B68A
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 5e 9a 46 20 a8 60 30 88-fa 2e c5 37 b7 29 0b 4e   ^.F .`0....7.).N
0010 - 41 67 2b b6 e7 8e 2e 12-8b 55 0c ad 59 80 f7 d5   Ag+......U..Y...
0020 - d1 07 8e fc 92 a1 2e 01-59 cf 00 2d d5 39 11 10   ........Y..-.9..
0030 - bf f3 89 af 2d 7a 02 59-49 54 3a bf e4 8b 97 f3   ....-z.YIT:.....
0040 - 55 da 4b 6f 9b 86 c4 85-eb e4 f9 a1 e3 74 76 be   U.Ko.........tv.
0050 - 65 57 76 ec e3 76 c9 c8-5a 47 c6 c2 ee eb bd ec   eWv..v..ZG......
0060 - 61 88 7c 35 8c a6 c0 b3-25 b5 79 06 99 df 66 75   a.|5....%.y...fu
0070 - 8e 9d 3a 17 61 40 7c 1c-09 e3 07 aa 49 b9 c3 cf   ..:.a@|.....I...
0080 - d7 ff 7d 1b cc 3f b9 3f-c7 bd ad 4d f9 4f eb 6c   ..}..?.?...M.O.l
0090 - 6f 42 2e c8 30 75 a9 07-d4 9e f0 12 6b 9c ca ac   oB..0u......k...
Start Time: 1544706461
Timeout   : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
Extended master secret: no
---
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>
closed

我也尝试了这个命令

curl -vL https://meteotravel.ru >/dev/null
* Rebuilt URL to: https://meteotravel.ru/
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 212.26.248.233...
* TCP_NODELAY set
* Connected to meteotravel.ru (212.26.248.233) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [222 bytes data]
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

我的nginx单独工作很好有配置

server {
access_log /var/log/nginx/default.access.log;
error_log /var/log/nginx/default.error.log warn;

listen 443 ssl default;
#listen 443 ssl http2 default reuseport;

# Redirect HTTP to HTTPS
if ($scheme = http) {
return 301 https://$host$request_uri;
}

ssl_certificate /etc/nginx/ssl/meteotravel.ru/mt.crt;
ssl_certificate_key /etc/nginx/ssl/meteotravel.ru/pk;
server_name meteotravel.ru;

root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
#   server_name _;
location /fop2{
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.

# Add index.php to the list if you are using PHP
try_files $uri $uri/ =404;
}
location /{
try_files $uri $uri/ =404;
}
}

错误在哪里？？

这是我在服务器上的haproxy配置

我还尝试为nginx设置proxy_procol：listen443 ssl proxy_procols；haproxy入口的选项：代理协议：[v1|v2|v2-ssl|v2-sls-cn]所有这些都是单独的并获取nginx后端日志中的错误

����kjih9876�����2.�.�*�&���=5"读取PROXY协议时，客户端：10.244.5.0，服务器：0.0.0.0:4432018/12/13 19:55:24〔错误〕7#7：2271断头："�ۆ�v+\��w�?���3.�Alm9i�� �L$&��h��0�,�(�$������kjih9876�����2.�.��&���=5"读取PROXY协议时，客户端：10.244.6.0，服务器：0.0.0.0:443

Answer 1

此问题包含注释CCD_ 1。它不需要，因为我们从haproxy-It tcp流中获取已经安全的数据。有了这个注释，我们进行了两次加密，nginx无法正确解密