User Tools

Site Tools


services:sentry

sentry

Summary

Sentry provides real-time job error tracking.

Self-hosted Sentry 25.10.0 running in OpenShift in the sentry namespace (migrated off the sentry.front VM 2026-09-08). See https://github.com/ceph/sepia-openshift/tree/main/sentry - MIGRATION.md and README.md there cover the architecture.

sentry.ceph.com is the canonical URL (GitHub OAuth/webhooks point at it), served through soko01 nginx + Anubis. /api/ and /extensions/ bypass Anubis so SDKs and webhooks work.

Images: getsentry stopped publishing to docker.io after 25.5.1 - 25.10.0+ is ghcr.io only. The sentry-build.yaml BuildConfig layers the sentry-nodestore-s3 plugin into the internal registry image. Nodestore is ODF RGW via OBC.

Admin Tasks

Upgrading

Walk the hard stops: next are 26.5.0 then 26.7.0. Don't skip them.

Troubleshooting

Check the logs

oc -n sentry get pods   # the usual suspects crash-loop visibly
oc -n sentry logs deploy/sentry-web

kafka/snuba/taskworker crash-looping

Check enableServiceLinks: false is on the pod - it's mandatory on every pod in the namespace. Services named kafka/clickhouse/redis make k8s inject KAFKA_PORT=tcp://... env vars that the images parse as config (snuba dies on int('tcp://...')).

taskbroker crash-looping after a rebuild

snuba bootstrap only creates snuba topics. The sentry-side topics (ingest-events, taskworker, taskworker-dlq, …) must be created by hand with kafka-topics --create.

Issues update but no events show

A failed nodestore write aborts save_event after the group update but before the kafka publish - postgres moves, clickhouse stays empty. Check the nodestore: the OBC configmap advertises port 443, but it must be http://$BUCKET_HOST:80.

post-process-forwarder crash-looping (rapidjson error)

A non-JSON message poison-pilled the events topic. Reset the post-process-forwarder consumer group offset past it.

400 DisallowedHost behind the proxy

The OpenShift router appends to X-Forwarded-Host by default. The route needs:

haproxy.router.openshift.io/set-forwarded-headers: if-none
services/sentry.txt · Last modified: by djgalloway