Differences

This shows you the differences between two versions of the page.

--- services:sentry [2016/12/06 19:53] – created dgalloway
+++ services:sentry [2026/10/02 18:58] (current) – djgalloway
@@ Line 1: / Line 1: @@
 ====== sentry ======
 ===== Summary =====
-Sentry provides real-time job error tracking.  See https://sentry.io/welcome/
+Sentry provides real-time job error tracking.
-The service is running on a VM in RHEV on sentry.front.sepia.ceph.com.  An nginx reverse proxy is set up on gw.sepia.ceph.com to allow HTTP access via ''sentry.ceph.com''.  Another nginx reverse proxy is set up on the sentry host to allow HTTP access via ''http://sentry.front.sepia.ceph.com''
+Self-hosted Sentry 25.10.0 running in OpenShift in the ''sentry'' namespace (migrated off the sentry.front VM 2026-09-08).  See https://github.com/ceph/sepia-openshift/tree/main/sentry - MIGRATION.md and README.md there cover the architecture.
+''sentry.ceph.com'' is the canonical URL (GitHub OAuth/webhooks point at it), served through soko01 nginx + Anubis.  ''/api/'' and ''/extensions/'' bypass Anubis so SDKs and webhooks work.
+Images: getsentry stopped publishing to docker.io after 25.5.1 - 25.10.0+ is **ghcr.io only**.  The ''sentry-build.yaml'' BuildConfig layers the ''sentry-nodestore-s3'' plugin into the internal registry image.  Nodestore is ODF RGW via OBC.
 ===== Admin Tasks =====
-==== Starting/Stopping Sentry ====
+==== Upgrading ====
-''supervisorctl start|stop sentry-worker sentry-web''
+Walk the hard stops: next are 26.5.0 then 26.7.0.  Don't skip them.
+===== Troubleshooting =====
+==== Check the logs ====
+<code>
+oc -n sentry get pods   # the usual suspects crash-loop visibly
+oc -n sentry logs deploy/sentry-web
+</code>
+==== kafka/snuba/taskworker crash-looping ====
+Check ''enableServiceLinks: false'' is on the pod - it's **mandatory** on every pod in the namespace.  Services named kafka/clickhouse/redis make k8s inject ''%%KAFKA_PORT=tcp://...%%'' env vars that the images parse as config (snuba dies on ''%%int('tcp://...')%%'').
+==== taskbroker crash-looping after a rebuild ====
+''snuba bootstrap'' only creates snuba topics.  The sentry-side topics (ingest-events, taskworker, taskworker-dlq, ...) must be created by hand with ''kafka-topics %%--%%create''.
+==== Issues update but no events show ====
+A failed nodestore write aborts save_event **after** the group update but **before** the kafka publish - postgres moves, clickhouse stays empty.  Check the nodestore: the OBC configmap advertises port 443, but it must be ''%%http://$BUCKET_HOST:80%%''.
-===== Notes =====
+==== post-process-forwarder crash-looping (rapidjson error) ====
+A non-JSON message poison-pilled the ''events'' topic.  Reset the ''post-process-forwarder'' consumer group offset past it.
-  * The Sentry application lives in ''/home/ubuntu/.virtualenvs/sentry''
+==== 400 DisallowedHost behind the proxy ====
-  * The Sentry conf lives in ''/home/ubuntu/.sentry/sentry.config.py''
+The OpenShift router **appends** to X-Forwarded-Host by default.  The route needs:
+<code>
+haproxy.router.openshift.io/set-forwarded-headers: if-none
+</code>