Differences

This shows you the differences between two versions of the page.

--- services:sentry [2021/05/03 14:46] – [Upgrade Sentry] djgalloway
+++ services:sentry [2026/10/02 18:58] (current) – djgalloway
@@ Line 1: / Line 1: @@
 ====== sentry ======
-This service was moved to a new VM in September 2020.
+===== Summary =====
+Sentry provides real-time job error tracking.
+Self-hosted Sentry 25.10.0 running in OpenShift in the ''sentry'' namespace (migrated off the sentry.front VM 2026-09-08).  See https://github.com/ceph/sepia-openshift/tree/main/sentry - MIGRATION.md and README.md there cover the architecture.
+''sentry.ceph.com'' is the canonical URL (GitHub OAuth/webhooks point at it), served through soko01 nginx + Anubis.  ''/api/'' and ''/extensions/'' bypass Anubis so SDKs and webhooks work.
+Images: getsentry stopped publishing to docker.io after 25.5.1 - 25.10.0+ is **ghcr.io only**.  The ''sentry-build.yaml'' BuildConfig layers the ''sentry-nodestore-s3'' plugin into the internal registry image.  Nodestore is ODF RGW via OBC.
 ===== Admin Tasks =====
-==== Reconfigure Sentry ====
+==== Upgrading ====
-If you need to change a setting in the sentry config,
+Walk the hard stops: next are 26.5.0 then 26.7.0.  Don't skip them.
+===== Troubleshooting =====
+==== Check the logs ====
 <code>
-ssh sentry.front.sepia.ceph.com
+oc -n sentry get pods   # the usual suspects crash-loop visibly
-sudo su - ubuntu
+oc -n sentry logs deploy/sentry-web
-cd sentry
-vim sentry/config.yml
-docker-compose down; docker-compose build; docker-compose up -d
 </code>
-==== Upgrade Sentry ====
+==== kafka/snuba/taskworker crash-looping ====
-<code>
+Check ''enableServiceLinks: false'' is on the pod - it's **mandatory** on every pod in the namespace.  Services named kafka/clickhouse/redis make k8s inject ''%%KAFKA_PORT=tcp://...%%'' env vars that the images parse as config (snuba dies on ''%%int('tcp://...')%%'').
-ssh sentry.front.sepia.ceph.com
-sudo su - ubuntu
-cd ~/sentry
-docker-compose down
-cd ~
-cp -r sentry sentry.$(date -u +%F).backup
-cd ~/sentry
-git pull
-./install.sh
-docker-compose up -d
-</code>
-===== DEPRECATED =====
+==== taskbroker crash-looping after a rebuild ====
-Sentry provides real-time job error tracking.  See https://sentry.io/welcome/
+''snuba bootstrap'' only creates snuba topics.  The sentry-side topics (ingest-events, taskworker, taskworker-dlq, ...) must be created by hand with ''kafka-topics %%--%%create''.
-The service is running on a VM in RHEV on sentry.front.sepia.ceph.com.  An nginx reverse proxy is set up on gw.sepia.ceph.com to allow HTTP access via ''sentry.ceph.com''.  Another nginx reverse proxy is set up on the sentry host to allow HTTP access via ''http://sentry.front.sepia.ceph.com''
+==== Issues update but no events show ====
+A failed nodestore write aborts save_event **after** the group update but **before** the kafka publish - postgres moves, clickhouse stays empty.  Check the nodestore: the OBC configmap advertises port 443, but it must be ''%%http://$BUCKET_HOST:80%%''.
-The sentry VM also runs the public kraken IRC bot.  See ''/home/ubuntu/helga''.
+==== post-process-forwarder crash-looping (rapidjson error) ====
+A non-JSON message poison-pilled the ''events'' topic.  Reset the ''post-process-forwarder'' consumer group offset past it.
-==== Admin Tasks ====
+==== 400 DisallowedHost behind the proxy ====
-=== Starting/Stopping Sentry ===
+The OpenShift router **appends** to X-Forwarded-Host by default.  The route needs:
-''supervisorctl start|stop sentry-worker sentry-web''
+<code>
+haproxy.router.openshift.io/set-forwarded-headers: if-none
-==== Notes ====
+</code>
-  * The Sentry application lives in ''/home/ubuntu/.virtualenvs/sentry''
-  * The Sentry conf lives in ''/home/ubuntu/.sentry/sentry.config.py''