为什么 Kubernetes HPA 没有缩减(内存)?

Question

摘要

在我们的 Kubernetes 集群中，我们引入了 HPA 内存和 CPU 限制。现在我们不明白为什么我们有一个服务的 2 个副本。

有问题的服务使用 57%/85% 内存，并且具有 2 个副本而不是一个副本。我们认为这是因为当您汇总两个 pod 的内存时，它超过 85%，但如果只有一个 pod，则不会。那么，这是否阻止了它缩小规模？我们能在这里做什么？

我们还观察到在部署服务时内存使用量达到峰值。我们正在 aks (azure( 中使用 spring-boot 服务，并认为它可以在那里向上扩展，永远不会向下扩展。我们错过了什么还是有人建议？

掌舵

HPA：

{{- $fullName := include "app.fullname" . -}}
{{- $ := include "app.fullname" . -}}
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: {{ $fullName }}-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "app.name" . }}
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 50
- type: Resource
resource:
name: memory
targetAverageUtilization: 85

在部署中：

# Horizontal-Pod-Auto-Scaler
resources:
requests:
memory: {{ $requestedMemory }}
cpu: {{ $requesteCpu }}
limits:
memory: {{ $limitMemory }}
cpu: {{ $limitCpu }}

使用服务默认值：

hpa:
resources:
request:
memory: 500Mi
cpu: 300m
limits:
memory: 1000Mi
cpu: 999m

Kubectl Get HPA -n dev

NAME                            REFERENCE                              TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
xxxxxxxx-load-for-cluster-hpa   Deployment/xxxxxxxx-load-for-cluster   34%/85%, 0%/50%   1         10        1          4d7h
xxx5-ccg-hpa                    Deployment/xxx5-ccg                    58%/85%, 0%/50%   1         10        1          4d12h
iotbootstrapping-service-hpa    Deployment/iotbootstrapping-service    54%/85%, 0%/50%   1         10        1          4d12h
mocks-hpa                       Deployment/mocks                       41%/85%, 0%/50%   1         10        1          4d12h
user-pairing-service-hpa        Deployment/user-pairing-service        41%/85%, 0%/50%   1         10        1          4d12h
aaa-registration-service-hpa    Deployment/aaa-registration-service    57%/85%, 0%/50%   1         10        2          4d12h
webshop-purchase-service-hpa    Deployment/webshop-purchase-service    41%/85%, 0%/50%   1         10        1          4d12h

kubectl describe hpa -n dev

Name:                                                     xxx-registration-service-hpa
Namespace:                                                dev
Labels:                                                   app.kubernetes.io/managed-by=Helm
Annotations:                                              meta.helm.sh/release-name: vwg-registration-service
meta.helm.sh/release-namespace: dev
CreationTimestamp:                                        Thu, 18 Jun 2020 22:50:27 +0200
Reference:                                                Deployment/xxx-registration-service
Metrics:                                                  ( current / target )
resource memory on pods  (as a percentage of request):  57% (303589376) / 85%
resource cpu on pods  (as a percentage of request):     0% (1m) / 50%
Min replicas:                                             1
Max replicas:                                             10
Deployment pods:                                          2 current / 2 desired
Conditions:
Type            Status  Reason              Message
----            ------  ------              -------
AbleToScale     True    ReadyForNewScale    recommended size matches current size
ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:           <none>

如果需要任何进一步的信息，请随时询问！

非常感谢您抽出宝贵时间！

干杯 知更鸟

Answer 1

确定所需副本计数的公式为：

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

对于您的问题来说，重要的部分是ceil[...]函数包装器：它总是向上舍入到下一个最接近的副本。 如果currentReplicas为 2，desiredMetricValue为 85%，则currentMetricValue必须为 42.5% 或更低才能触发缩减。

在您的示例中，currentMetricValue为 57%，因此您可以得到

desiredReplicas = ceil[2 * (57 / 85)]
= ceil[2 * 0.671]
= ceil[1.341]
= 2

你是对的，如果currentReplicas是1，HPA也不会觉得有必要扩大规模;实际利用率需要攀升到85%以上才能触发它。