我上周在 kubernetes 上部署了哨兵,使用这个掌舵图稳定/哨兵。
Pod 工作正常,但我无法访问该网站,每次我访问端点时它都会崩溃。
我检查了工人,哨兵网和邮政豆荚的日志,看到了这个。
这是网站窗格的日志:
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 396, in execute
return real_execute(self, sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute
return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.ProgrammingError: ProgrammingError('relation "sentry_projectkey" does not existnLINE 1: ...te_limit_window", "sentry_projectkey"."data" FROM "sentry_pr...n ^n',)
SQL: SELECT "sentry_projectkey"."id", "sentry_projectkey"."project_id", "sentry_projectkey"."label", "sentry_projectkey"."public_key", "sentry_projectkey"."secret_key", "sentry_projectkey"."roles", "sentry_projectkey"."status", "sentry_projectkey"."date_added", "sentry_projectkey"."rate_limit_count", "sentry_projectkey"."rate_limit_window", "sentry_projectkey"."data" FROM "sentry_projectkey" WHERE ("sentry_projectkey"."project_id" = %s AND "sentry_projectkey"."roles" = "sentry_projectkey"."roles" | 1) LIMIT 1
这是 postgres pod 的日志:
ERROR: relation "sentry_option" does not exist at character 114
STATEMENT: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= '2019-09-13 14:02:49.793126+00:00'
ERROR: relation "sentry_monitor" does not exist at character 328
STATEMENT: SELECT "sentry_monitor"."id", "sentry_monitor"."guid", "sentry_monitor"."organization_id", "sentry_monitor"."project_id", "sentry_monitor"."name", "sentry_monitor"."status", "sentry_monitor"."type", "sentry_monitor"."config", "sentry_monitor"."next_checkin", "sentry_monitor"."last_checkin", "sentry_monitor"."date_added" FROM "sentry_monitor" WHERE ("sentry_monitor"."next_checkin" < '2019-09-13 15:02:50.996171+00:00' AND "sentry_monitor"."type" IN (2, 3) AND NOT ("sentry_monitor"."status" IN (1, 2, 3))) LIMIT 10000
ERROR: relation "sentry_scheduledjob" does not exist at character 179
STATEMENT: SELECT "sentry_scheduledjob"."id", "sentry_scheduledjob"."name", "sentry_scheduledjob"."payload", "sentry_scheduledjob"."date_added", "sentry_scheduledjob"."date_scheduled" FROM "sentry_scheduledjob" WHERE "sentry_scheduledjob"."date_scheduled" <= '2019-09-13 15:02:51.984402+00:00' LIMIT 101
ERROR: relation "sentry_authidentity" does not exist at character 288
STATEMENT: SELECT "sentry_authidentity"."id", "sentry_authidentity"."user_id", "sentry_authidentity"."auth_provider_id", "sentry_authidentity"."ident", "sentry_authidentity"."data", "sentry_authidentity"."last_verified", "sentry_authidentity"."last_synced", "sentry_authidentity"."date_added" FROM "sentry_authidentity" WHERE "sentry_authidentity"."last_synced" <= '2019-09-13 14:02:56.685172+00:00'
和工作器豆荚:
result = func(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/tasks/options.py", line 36, in sync_options
for option in Option.objects.filter(last_updated__gte=cutoff_dt).iterator():
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 396, in execute
return real_execute(self, sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute
return self.cursor.execute(sql, clean_bad_params(params))
ProgrammingError: ProgrammingError('relation "sentry_option" does not existnLINE 1: ...ion"."value", "sentry_option"."last_updated" FROM "sentry_op...n ^n',)
SQL: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= %s
15:09:24 [ERROR] celery.worker.job: Task sentry.tasks.options.sync_options[e1684c22-6593-48d9-94c7-2c47ee7fc397] raised unexpected: ProgrammingError(u'ProgrammingError('relation "sentry_option" does not exist\nLINE 1: ...ion"."value", "sentry_option"."last_updated" FROM "sentry_op...\n ^\n',)nSQL: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= %s ',) (data={u'hostname': 'celery@teq-sentry-worker-6ddfb9658c-gbkth', u'name': 'sentry.tasks.options.sync_options', u'args': '[]', u'internal': False, u'kwargs': '{}', u'id': 'e1684c22-6593-48d9-94c7-2c47ee7fc397'})
您必须手动迁移架构, 运行以下命令以迁移架构:
kubectl exec -it <your-pod-name> sentry upgrade
求解。 谢谢大家的帮助。问题是我没有足够的 RAM 来运行迁移脚本,所以我将 VPS 升级到 8GB RAM,然后它就可以工作了。
由于迁移脚本有一些内存泄漏,因此 Pod 至少需要 4GB 可用内存才能运行。在 Github 上查看此评论
面临类似的问题。在我们的例子中,我们已经启用了入口(Nginx(。并且此设置与service.type冲突:LoadBalancer
,只需将其更改为ClusterIP
并再次执行清除设置即可。
TLDR - DB init 在"安装后"事件(所有部署的组件都处于就绪状态(时启动,但在 LoadBallancer、入口和--wait
组合的情况下,Web 服务必须分配 IP (https://github.com/helm/helm/blob/master/docs/using_helm.md#helpful-options-for-installupgraderollback(。由于某种原因,它不是真的,Web 服务总是返回false
.将服务设置从LoadBalancer
更改为ClusterIP
后,问题消失了。