在 kubernetes 集群中部署哨兵掌舵



我上周在 kubernetes 上部署了哨兵,使用这个掌舵图稳定/哨兵。

Pod 工作正常,但我无法访问该网站,每次我访问端点时它都会崩溃。

我检查了工人,哨兵网和邮政豆荚的日志,看到了这个。

这是网站窗格的日志:

self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 396, in execute
return real_execute(self, sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute
return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.ProgrammingError: ProgrammingError('relation "sentry_projectkey" does not existnLINE 1: ...te_limit_window", "sentry_projectkey"."data" FROM "sentry_pr...n                                                             ^n',)
SQL: SELECT "sentry_projectkey"."id", "sentry_projectkey"."project_id", "sentry_projectkey"."label", "sentry_projectkey"."public_key", "sentry_projectkey"."secret_key", "sentry_projectkey"."roles", "sentry_projectkey"."status", "sentry_projectkey"."date_added", "sentry_projectkey"."rate_limit_count", "sentry_projectkey"."rate_limit_window", "sentry_projectkey"."data" FROM "sentry_projectkey" WHERE ("sentry_projectkey"."project_id" = %s  AND "sentry_projectkey"."roles" =  "sentry_projectkey"."roles" | 1) LIMIT 1

这是 postgres pod 的日志:

ERROR:  relation "sentry_option" does not exist at character 114
STATEMENT:  SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= '2019-09-13 14:02:49.793126+00:00' 
ERROR:  relation "sentry_monitor" does not exist at character 328
STATEMENT:  SELECT "sentry_monitor"."id", "sentry_monitor"."guid", "sentry_monitor"."organization_id", "sentry_monitor"."project_id", "sentry_monitor"."name", "sentry_monitor"."status", "sentry_monitor"."type", "sentry_monitor"."config", "sentry_monitor"."next_checkin", "sentry_monitor"."last_checkin", "sentry_monitor"."date_added" FROM "sentry_monitor" WHERE ("sentry_monitor"."next_checkin" < '2019-09-13 15:02:50.996171+00:00'  AND "sentry_monitor"."type" IN (2, 3) AND NOT ("sentry_monitor"."status" IN (1, 2, 3))) LIMIT 10000
ERROR:  relation "sentry_scheduledjob" does not exist at character 179
STATEMENT:  SELECT "sentry_scheduledjob"."id", "sentry_scheduledjob"."name", "sentry_scheduledjob"."payload", "sentry_scheduledjob"."date_added", "sentry_scheduledjob"."date_scheduled" FROM "sentry_scheduledjob" WHERE "sentry_scheduledjob"."date_scheduled" <= '2019-09-13 15:02:51.984402+00:00'  LIMIT 101
ERROR:  relation "sentry_authidentity" does not exist at character 288
STATEMENT:  SELECT "sentry_authidentity"."id", "sentry_authidentity"."user_id", "sentry_authidentity"."auth_provider_id", "sentry_authidentity"."ident", "sentry_authidentity"."data", "sentry_authidentity"."last_verified", "sentry_authidentity"."last_synced", "sentry_authidentity"."date_added" FROM "sentry_authidentity" WHERE "sentry_authidentity"."last_synced" <= '2019-09-13 14:02:56.685172+00:00' 

和工作器豆荚:

result = func(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/tasks/options.py", line 36, in sync_options
for option in Option.objects.filter(last_updated__gte=cutoff_dt).iterator():
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 396, in execute
return real_execute(self, sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
return func(self, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
return func(self, sql, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute
return self.cursor.execute(sql, clean_bad_params(params))
ProgrammingError: ProgrammingError('relation "sentry_option" does not existnLINE 1: ...ion"."value", "sentry_option"."last_updated" FROM "sentry_op...n                                                             ^n',)
SQL: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= %s 
15:09:24 [ERROR] celery.worker.job: Task sentry.tasks.options.sync_options[e1684c22-6593-48d9-94c7-2c47ee7fc397] raised unexpected: ProgrammingError(u'ProgrammingError('relation "sentry_option" does not exist\nLINE 1: ...ion"."value", "sentry_option"."last_updated" FROM "sentry_op...\n                                                             ^\n',)nSQL: SELECT "sentry_option"."id", "sentry_option"."key", "sentry_option"."value", "sentry_option"."last_updated" FROM "sentry_option" WHERE "sentry_option"."last_updated" >= %s ',) (data={u'hostname': 'celery@teq-sentry-worker-6ddfb9658c-gbkth', u'name': 'sentry.tasks.options.sync_options', u'args': '[]', u'internal': False, u'kwargs': '{}', u'id': 'e1684c22-6593-48d9-94c7-2c47ee7fc397'})

您必须手动迁移架构, 运行以下命令以迁移架构:

kubectl exec -it <your-pod-name> sentry upgrade

求解。 谢谢大家的帮助。问题是我没有足够的 RAM 来运行迁移脚本,所以我将 VPS 升级到 8GB RAM,然后它就可以工作了。

由于迁移脚本有一些内存泄漏,因此 Pod 至少需要 4GB 可用内存才能运行。在 Github 上查看此评论

面临类似的问题。在我们的例子中,我们已经启用了入口(Nginx(。并且此设置与service.type冲突:LoadBalancer,只需将其更改为ClusterIP并再次执行清除设置即可。

TLDR - DB init 在"安装后"事件(所有部署的组件都处于就绪状态(时启动,但在 LoadBallancer、入口和--wait组合的情况下,Web 服务必须分配 IP (https://github.com/helm/helm/blob/master/docs/using_helm.md#helpful-options-for-installupgraderollback(。由于某种原因,它不是真的,Web 服务总是返回false.将服务设置从LoadBalancer更改为ClusterIP后,问题消失了。

相关内容

  • 没有找到相关文章

最新更新