What runs when, how the operator learns that something is wrong and where to look first.

Reference Value
Subject and boundary Long-running services, scheduled jobs, health signals, alerting and safe diagnosis. Repairs are on ‣.
Verified source docker-compose.yml, pipeline-control.sh, .github/workflows, freshness-probe.sh, alert-issue.sh, health/app.py, worker_starts.py, worker_start_probe.py, worker_watchdog.py, health/checks.py at main 9733eb26d92533f56cdc8bc46b81b93fbb5987c8, checked November 10, 2026.
Support state Implemented source and tracked configuration. This page does not report current production health; use the published data-health payload, the freshness probe runs and open alert issues for that.

Long-running services (VM, Docker Compose)

Service Command Notes
postgres PostgreSQL 16 + PostGIS image Host port bound to 127.0.0.1 by default (POSTGRES_BIND_ADDRESS)
worker run-realtime-worker stm (image default) Ticks every 10 s aligned to UTC, because STM vehicle positions are phased; each endpoint is captured on its own cadence (see ‣). Capture, Silver, Gold, then best-effort live publish of changed files. A hang watchdog (REALTIME_WATCHDOG_SECONDS, 600) writes every thread's stack to the container log and exits the process; restart: unless-stopped starts it again. Honors PIPELINE_PAUSED. Each start increments core.worker_starts through a recorder subprocess bounded to five seconds; a recorder failure logs one warning and capture continues
pruner run-pruner-loop stm Silver and Gold retention every PRUNER_SLEEP_SECONDS (15); database only, no feed or storage secrets
health uvicorn transit_ops.health.app:app on 8080 /health/live liveness and /health report (database, Bronze, pipeline age up to HEALTH_MAX_PIPELINE_AGE_SECONDS = 900, provider live policy)
caddy Reverse proxy to health:8080 Bound to 127.0.0.1:8080 and :8443 by default; TLS only when CADDY_SITE_ADDRESS is set

Each service sets PGAPPNAME (transit-worker, transit-pruner, transit-octranspo-worker, transit-octranspo-pruner, transit-health), so connections are identifiable in pg_stat_activity. The label is not an identity check. The tracked Compose file also defines octranspo-worker (extends worker, runs run-realtime-worker octranspo on a 10-second start-to-start worker tick, with unphased vehicle captures every 20 seconds and trip updates every 30 seconds, gets OCTRANSPO_API_KEY with the STM key blank) and octranspo-pruner (extends pruner, runs run-pruner-loop octranspo). Production runs both from an untracked Compose override on the VM; no workflow deploys the VM. See Decisions and limitations.

Scheduled jobs (GitHub Actions, UTC)

Workflow Schedule What it does
Daily Static Pipeline 06:00 daily init-db, seed-core, run-static-pipeline per active provider, then publish-all --tier static only if every provider succeeded
Daily Warm Rollups 07:00 daily Migrate, sync alert archive, build-warm-rollups, then retention (prune-i3-storage, prune-warm-rollup-storage, prune-bronze-storage, retention-proof-report)
Daily Warm Rollups (Wednesday) 07:00 Wednesday Adds gated publish-all --tier historic and verify-historic-publish proof. A manual run with no Bronze catch-up also publishes
Freshness Probe Every 15 min; backup check hourly at :07 Six freshness checks per provider; verify-backup-freshness
Worker Start Probe Every 15 min Read-only: role fingerprint, migration privilege preflight, current Alembic revision and core.worker_starts counts; a count above the issue marker opens or comments on the provider's worker-restart-alert issue
Historic snapshot GC marks 10:23 Monday gc-historic-snapshots --mode mark; apply mode is disabled
Weekly pg_repack 08:00 Sunday Bloat check in dry-run mode
Refresh Basemap Extract 06:00 on the 1st Rebuilds provider basemap extracts in R2
secret-scan 10:17 Monday gitleaks over the repository
Public Site Probe :04, :19, :34, :49 each hour Web health/version, both home documents and the KPI v1 response without ?provider (the STM default); failures are retried once after 5 seconds
Public Site Probe, weekly 10:37 Monday Read-only links from the bounded yesid.dev sitemap to Transit, and six Transit EN/FR home/status/map pages to yesid.dev; no contact form submission

The Warm workflow delegates its prepare, rollups, publish, handoff, proof and retention policy to transit-ops daily-run; daily_run.py owns receipts and daily_run_process.py owns execution and cancellation. Inspect attempt-scoped artifacts before diagnosing a stage failure.

Warm, Historic Publish Recovery and GC share the daily-warm-rollups concurrency group, so they never run at the same time.

Signals and alerting

Signal Detects
Freshness check A Live manifest files.live.generated_utc older than the limit
Check B Static schedule not observed within 36 hours
Check C No successful historic publication within 7 days plus 36 hours
Check D Oldest enabled realtime endpoint capture too old in the database
Check E Burst of failed raw.ingestion_runs, including silver_load failures
Check F Newest Silver realtime snapshot too old while raw capture continues
Backup freshness Newest logical backup missing or stale
Worker start count A realtime worker start, including a deliberate restart, a rollout or a watchdog exit; not the exit code or cause. A watchdog exit leaves a Timeout (...)! line and thread stacks in the container log
Live policy (/health) A degraded component <provider>_live_policy_<route> for each line declared in lines_without_live_data that has ever been seen with live data: the manifest is outdated. The check has no time window, so it keeps reporting until the declaration changes

Alerts are labeled GitHub issues created by alert-issue.sh (label pipeline-alert unless ALERT_LABEL sets another, such as worker-restart-alert): an open issue means alerting, closing it means recovered, and an issue already open is not duplicated. The Daily Static and Warm workflows also update their own status issues.

alert-issue.sh mentions @mgkdante in the initial issue body and attempts assignment after creation. Assignment failure preserves the issue and mention; dedupe leaves an existing issue untouched. Recovery comments and closes. The workflows grant issues: write. GitHub notification preferences determine email destination and receipt. Worker-start issues #645 and #646 reached the owner's inbox on November 6, 2026; comments on an already open issue and the other labels have no observed inbox receipt. Creating an issue is not proof of email delivery.

Production runtime

Observed November 9, 2026 to November 10, 2026; this is a dated record, not a live status.

Item Observation
Realtime workers OC octranspo-worker runs transit-worker:e2779026, started November 10, 2026 9:20 PM for #748, merge e2779026. It repairs the effective cadence after #746’s configuration rollout; core metadata from #746 is unchanged. No migration. STM remains on transit-worker:ed25527f, started at 17:53:03Z after #742. Pruners, health, Postgres and Caddy were not recreated. Detailed rollout/rollback and validation evidence is owned by ‣.
OC effective cadence Verified after the e2779026 rollout: raw vehicle captures match the 20 s target, with trip updates still at 30 s and source lag still 60 s. All observed HTTP, Gold and publication results succeeded; no unexpected OC restart. Exact interval samples, the observation window and its limits are owned by ‣.
Pruners and health API Image bcd815de
Live age After the 2026-11-09 redeploy, Live age had a median of 19 s for STM and 25 s for OC Transpo. Samples were taken during this period only; they are not a service-level guarantee
Database Revision 0096_provider_live_policy, unchanged by #742 (see ‣)

Pause and resume

apps/db/scripts/pause-pipeline.sh disables the Daily Static and Daily Warm workflows and stops the Compose worker and octranspo-worker through apps/db/infra/pipeline-control.sh; apps/db/scripts/resume-pipeline.sh reverses both. The pruners are not paused. The production VM's script and override require stopping and starting octranspo-worker by name. They change production schedules and services, so they are operator actions.

Diagnose before mutating

Compare capture age, measurement age, derivative completion and publication generation separately; they fail independently. Start from the published status/data_health.json, probe results, open alert issues and recorded job output. Source configuration and local test passes never prove production health.