Which checks exist, what each one proves and how CI runs them on every pull request.
| Reference | Value |
|---|---|
| Subject and boundary | Local checks per app, the CI workflows that gate merges and the limits of each kind of evidence. Environment setup is on ‣. |
| Verified source | CONTRIBUTING.md, ci.yml, web.yml, secret-scan.yml at main 9733eb26d92533f56cdc8bc46b81b93fbb5987c8, with web.yml, its scripts and the e2e config re-checked at main 4fcc6c01 and ruleset 17834816 read on 2026-11-09; checked November 10, 2026, with the layout matrix in playwright.config.ts. CI rows re-checked on main 9733eb26 and ruleset 17834816 read on November 10, 2026, after #723, #725, #726, #727, #729, #730, #731 and #733. |
| Support state | Implemented and run by GitHub Actions on pull requests and pushes to main. |
bun run test:commands
bun run --cwd apps/data-proxy check
bun run --cwd apps/data-proxy test
bun run --cwd apps/web check
bun run --cwd apps/web lint
bun run --cwd apps/web format:check
bun run --cwd apps/web test
bun run --cwd apps/web build
Browser tests: from apps/web, install Chromium once with bunx playwright install chromium, then bun run test:e2e (builds first). Fast local loop: from the repository root, bun run test:changed (a root package.json script, not an apps/web one) runs Vitest --changed against origin/main plus the impacted Playwright specs at one phone and one desktop project (#727); it is a local accelerator, not a CI gate. Python: from apps/db, uv run ruff check src tests, uv run mypy and uv run pytest tests. Strict mypy scope is configured in pyproject.toml.
| Check | Proves | Does not prove |
|---|---|---|
Reader kernel tests (tests/test_gold_reader.py) |
Units, nulls, rounding and window math | Real SQL execution or complete feeds |
*_real_db.py tests |
Transactions, constraints, DST and capture-day behavior on PostgreSQL | Production data or other hosts; they skip without a disposable target, and a skip is not a pass |
apps/data-proxy/test |
Routing, conditional and range responses, KPI freshness with local R2 and Cache API substitutes | Live R2 availability |
generated-contract.test.ts, handwritten-conformance.test.ts |
Web schemas match the exported contract | Numerical correctness of the values |
Vitest component tests; Playwright e2e in 3 shards, with request-only specs run once in an http project; the layout grid, 2,534 cases on 12 duration-balanced shards in the pinned Playwright container (#726, #729), across 16 layout-* projects: Chromium at 360×800, 390×844, 768×1024, 1280×900, 1440×900, 1920×1080 and 844×390; WebKit at 1440×900, 390×844, 844×390 and 1280×900; light everywhere, dark at 390 and 1440 wide (and 1280 in Chromium); e2e/layout.spec.ts loops English and French. A layout failure keeps a lean trace without DOM snapshots, screenshots or sources (#733) |
Rendered behavior, keyboard flows, axe checks in Chromium | Every browser, physical device or assistive technology |
test:b9-display |
Displayed values match an independent oracle for selected surfaces | Surfaces outside its fixtures |
web-build-artifact.test.mjs |
Sealing and restore rejection rules for the deploy artifact | That a given deployment serves those bytes |
| controlCohesion.test.ts | No hand-built role="tab"/tablist and no <button> with a pill, chip or tab class under the map, stops, lines, trips and network features or components/surface; parsed with the Svelte compiler, empty allowlist |
That the shared controls themselves behave correctly |
| test_explicit_provider_ids.py | No provider_id parameter or field default anywhere in the snapshots package (syntax-tree walk), and publish-providers requires --default-provider |
Provider correctness of values passed at runtime |
Use literal counterexamples for material math. A test that derives its expectation through the implementation can repeat the implementation's mistake.
| Workflow | Runs on | Jobs |
|---|---|---|
CI (ci.yml) |
PRs, merge queue (merge_group) and pushes to main; path classification skips unaffected jobs |
Offline pytest, Ruff, strict mypy; single Alembic head; real-database suite on a postgis/postgis:16-3.4 service after a full alembic upgrade head replay, run with 4 pytest-xdist workers, each on its own database copied from the template with its database settings (#723); runtime-image verification runs in the offline job; one required-context summary |
web (web.yml) |
PRs, merge queue, pushes to main, manual |
Proxy syntax, tests and Wrangler dry run; deploy-scope and basemap contracts; OG, icon and poster drift gates; lint, format, design lint, type-check; build and seal; Wrangler dry run; authenticated Chromium; B9 display check; unit tests. classify runs web-ci-paths.mjs from the base revision and outputs run_e2e and run_layout; prerelease-guard; e2e in 3 shards, skipped only for docs-only PRs; layout in 12 duration-balanced shards with 2 workers each in the pinned Playwright container (#726, #729), on pull requests only, skipped when run_layout is false; the always-run ci aggregate; then a preview, or on main a production deploy, followed by the full layout grid on 12 shards against the deployed SHA (#725, #729). On a push to main, queue-gate (#741) looks for the exact-SHA, same-workflow, first-attempt merge_group run with every required job green and verifies its sealed artifact; when proven, ci-work, unit and e2e are skipped and that artifact deploys. Otherwise the push runs build, unit and core e2e before deploying |
secret-scan |
PRs to and pushes on main, merge queue (merge_group), weekly Monday 10:17 UTC, manual |
gitleaks with .gitleaks.toml; scheduled failures open an alert issue |
Ruleset 17834816 on main (read 2026-11-10) requires gitleaks, ci, offline-tests, alembic-single-head and real-db-tests from GitHub Actions (strict up-to-date off) plus CodeQL code scanning results, and merges through a merge queue: squash, ALLGREEN, up to 5 entries built and merged per group, no wait, 60-minute check timeout. Admins bypass always. Since #730, queue entries after the first trust main's policy instead of failing on a base that is not yet on main. Org ruleset 26919834 also protects the default branch (no deletion, no force push, changes through a pull request).
Measured November 10, 2026: real-DB job 4m39s active against 11m23s, pytest 234.3 s against 583.6 s (job 129763552660). Merge to live 6m23s in run 42784846952 (the deploy promoted the merge queue's sealed artifact; unit tests re-run on push take 323 s of it) (n=1) against p50 11.2 and p90 20.9 min; its post-deploy grid finished 25 min after the merge. The PR decision got slower: p50 18.7 min and p90 19.7 min over 13 green heads, against 11.1 and 21.0 before, because the container layout shards take 10.7 to 14.9 min on 4 shards. The 12-shard split (#729) merged at 11:46 UTC; its own PR run reached the decision in 8m16s (n=1, run 42808245036), with Playwright time per shard 206 to 312 s against a 270 s target. Deployment jobs are described on Deployment and release.
Local, CI, built artifact, deployed and sustained operation are separate evidence. A green build or preview does not show that production serves that source, that feeds are healthy, or that the service stays available.
| Boundary | Maintained checks | Evidence limit |
|---|---|---|
| KPI v1 | apps/data-proxy/test/kpis-contract.test.mjs, kpis.test.mjs, contracts/kpis-v1.examples.json |
Shape, null lanes, cache crossing/cold-isolate behavior; not browser hold state |
| Public probes | .github/scripts/site-probe.test.mjs, site-probe-weekly.test.mjs, site-probe-non-sending.test.mjs |
Local substitutes and no-send invariants; not real alert email receipt or a future scheduled run |
| Rider health | apps/db/tests/test_live_serving_capture.py, affected builder/real-DB regressions, apps/web/src/lib/v1/live/store.svelte.test.ts, map selectionGrace.svelte.test.ts and health-presentation tests |
Accepted provenance, payloads without capture timestamps, explicit unknown, retry retention and EN/FR copy; not VM rollout or a real production partial stall |
| Reliability bounds | apps/web/src/lib/v1/verdict.test.ts, cross-surface consistency and B9 stop/network oracles |
Exact counts, published ranges and literal boundary counterexamples |
| Shared design contract | e2e/design-contracts.spec.ts, unit/SSR and browser suites |
Real Transit callers on v0.40.0 (#738); not physical devices or AT |
| Worker start alerts | apps/db/tests/test_worker_starts.py, test_worker_start_probe.py, test_workflow_alerting_contract.py |
Bounded recorder, shutdown during observation, read-only probe, migration grants, marker dedupe and issue creation or comment, with fakes and disposable real-database cases; not a real crash, Docker exit code or email receipt |
| Share cards | apps/web/src/lib/og/card.test.ts, src/lib/seo/share-copy.test.ts, routeSeo.test.ts, e2e/head-ssr.spec.ts, og:check |
Card layout, copy per destination, image selection and SSR head tags; not how a social platform renders or caches a card |
| Remembered content sections | remembered-sections.spec.ts and SectionsToggle.svelte.test.ts |
First-visit expansion with Collapse all (seeded collapsed where a case needs closed content), bulk-choice memory across pages/reload, readable charts/departures on opening, polling while collapsed, independent controls and map exclusion; source-backed coverage; execution evidence belongs to its specific run |
CI uses repository-owned .github/scripts/ci-paths.mjs with classifier tests and pinned Node. Unknown paths select DB checks. Recognized unrelated paths may skip DB work while required-context summaries pass; root GLOSSARY.md is an unknown path, so its documentation PR runs the DB jobs too. Transit's secret scanner is repository-owned. Bun comes from .bun-version, raised in #731 because the earlier resolver could stall on tarball installs. The exact retry sequence is source/test evidence; it has not been reproduced in a real production browser.