Differences

This shows you the differences between two versions of the page.

--- services:sentry [2020/09/11 22:34] – djgalloway
+++ services:sentry [2026/10/02 18:58] (current) – djgalloway
@@ Line 1: / Line 1: @@
 ====== sentry ======
-This service was moved to a new VM in September 2020.
+===== Summary =====
+Sentry provides real-time job error tracking.
+Self-hosted Sentry 25.10.0 running in OpenShift in the ''sentry'' namespace (migrated off the sentry.front VM 2026-09-08).  See https://github.com/ceph/sepia-openshift/tree/main/sentry - MIGRATION.md and README.md there cover the architecture.
+''sentry.ceph.com'' is the canonical URL (GitHub OAuth/webhooks point at it), served through soko01 nginx + Anubis.  ''/api/'' and ''/extensions/'' bypass Anubis so SDKs and webhooks work.
+Images: getsentry stopped publishing to docker.io after 25.5.1 - 25.10.0+ is **ghcr.io only**.  The ''sentry-build.yaml'' BuildConfig layers the ''sentry-nodestore-s3'' plugin into the internal registry image.  Nodestore is ODF RGW via OBC.
 ===== Admin Tasks =====
-==== Reconfigure Sentry ====
+==== Upgrading ====
-If you need to change a setting in the sentry config,
+Walk the hard stops: next are 26.5.0 then 26.7.0.  Don't skip them.
+===== Troubleshooting =====
+==== Check the logs ====
 <code>
-ssh sentry.front.sepia.ceph.com
+oc -n sentry get pods   # the usual suspects crash-loop visibly
-sudo su - ubuntu
+oc -n sentry logs deploy/sentry-web
-cd sentry
-vim sentry/config.yml
-docker-compose down; docker build; docker-compose up -d
 </code>
-===== DEPRECATED =====
+==== kafka/snuba/taskworker crash-looping ====
-Sentry provides real-time job error tracking.  See https://sentry.io/welcome/
+Check ''enableServiceLinks: false'' is on the pod - it's **mandatory** on every pod in the namespace.  Services named kafka/clickhouse/redis make k8s inject ''%%KAFKA_PORT=tcp://...%%'' env vars that the images parse as config (snuba dies on ''%%int('tcp://...')%%'').
-The service is running on a VM in RHEV on sentry.front.sepia.ceph.com.  An nginx reverse proxy is set up on gw.sepia.ceph.com to allow HTTP access via ''sentry.ceph.com''.  Another nginx reverse proxy is set up on the sentry host to allow HTTP access via ''http://sentry.front.sepia.ceph.com''
+==== taskbroker crash-looping after a rebuild ====
+''snuba bootstrap'' only creates snuba topics.  The sentry-side topics (ingest-events, taskworker, taskworker-dlq, ...) must be created by hand with ''kafka-topics %%--%%create''.
-The sentry VM also runs the public kraken IRC bot.  See ''/home/ubuntu/helga''.
+==== Issues update but no events show ====
+A failed nodestore write aborts save_event **after** the group update but **before** the kafka publish - postgres moves, clickhouse stays empty.  Check the nodestore: the OBC configmap advertises port 443, but it must be ''%%http://$BUCKET_HOST:80%%''.
-==== Admin Tasks ====
+==== post-process-forwarder crash-looping (rapidjson error) ====
-=== Starting/Stopping Sentry ===
+A non-JSON message poison-pilled the ''events'' topic.  Reset the ''post-process-forwarder'' consumer group offset past it.
-''supervisorctl start|stop sentry-worker sentry-web''
-==== Notes ====
+==== 400 DisallowedHost behind the proxy ====
+The OpenShift router **appends** to X-Forwarded-Host by default.  The route needs:
-  * The Sentry application lives in ''/home/ubuntu/.virtualenvs/sentry''
+<code>
-  * The Sentry conf lives in ''/home/ubuntu/.sentry/sentry.config.py''
+haproxy.router.openshift.io/set-forwarded-headers: if-none
+</code>