test: make the suite's assertions falsifiable and its state isolated

Several assertions could not fail:

- "the catch update remains saved" was `caught.isChecked() || notes.includes(...)`
  shared by two scenarios, so either half satisfied both. Split into two steps
  that each assert the outcome their own scenario is about.
- The token-refresh check read a global counter with `> 0` and asserted an upload
  had happened `some(...)`, both already satisfied by the preceding scenario. It
  now asserts exactly one refresh, ordered before the upload.
- The box step ignored its box argument and asserted on the first N entries on
  the page; it now scopes to that box and checks its full contents.
- The filter step asserted on whichever entry was first after filtering; it now
  records the caught entry beforehand and names it, and checks the filter did not
  exclude everything.
- The empty-state precondition asserted emptiness instead of establishing it,
  which a fresh user satisfies for free.
- Offline coverage was `caches.keys().length > 0`. It now checks the precache
  contract: one workbox cache holding the shell and a revisioned web manifest,
  with _app/immutable assets cached without a revision query. The scenario that
  claimed to test a trailing slash did not; it is replaced with real offline
  client-side navigation.

The mock provider kept recorded requests, its refresh counter and the
fail-uploads switch in one process-wide object that only one step reset, so
scenario order was load-bearing and the failing-upload scenario poisoned
everything after it. An auto fixture now resets it per scenario, and the mock no
longer records its own control-plane calls. That reset is why the suite stays on
a single worker, which is now documented.

Coverage was gated at 90% per file over an allowlist of exactly the five files
that had tests, so new code was invisible to it permanently. It now measures all
of src/lib with global thresholds at the measured baseline, and no longer runs
the unit tests twice.

Also: a global teardown removes the users each run creates, the Supabase wrapper
distinguishes a stopped stack from a broken CLI call and detects an unseeded
database, the sign-in rate limit is raised above what one serial run needs, and
the integration suite no longer falls back to a hard-coded anon key that would
mask a misconfigured run.

The password-reset scenarios are renamed to what they actually cover: following a
real recovery link bounces to /signin, because the browser client persists no
cookies and so cannot keep the session it parses out of the URL. The helper for
the real flow is left in place and the gap is documented.
This commit is contained in:
Josh Creek
2026-09-13 17:36:36 +01:00
parent 3a2c18bbeb
commit 4af33709a3
27 changed files with 406 additions and 102 deletions
+28 -7
View File
@@ -42,15 +42,21 @@ The test suite is split by responsibility so a failure points to the correct lay
definitions and browser fixtures live beside it under `tests/bdd`.
- `tests/build` verifies generated service-worker and manifest artifacts after each supported build.
Run the offline suites while developing:
Run the offline suites while developing. `test:fast` includes the coverage run, so there is no need
to run both:
```bash
npm run test:fast
npm run test:coverage
```
Database and BDD tests require Docker and the local Supabase stack. The wrappers read local keys from
`supabase status`; no credentials are written to disk or committed:
Coverage is measured across all of `src/lib` (excluding type-only models and the browser-only
store/action modules) with global thresholds set to the current baseline, so new untested code lowers
the number instead of being invisible to the gate. Ratchet the thresholds in `vitest.config.mts` up as
coverage grows, never down.
Database and BDD tests require Docker and the local Supabase stack. The wrappers read the local keys
from `supabase status` at run time, so no keys are hard-coded in the test files; the local stack's
well-known demo keys do appear in `.env.local.example`, and no real credentials are committed:
```bash
npm run supabase:start
@@ -67,9 +73,24 @@ Gherkin describes outcomes in domain language. Keep selectors, API calls, test-u
provider mocks in step definitions or support fixtures. `@product-review` marks a rule that should be
reviewed with product stakeholders, but does not skip it. Missing or ambiguous steps fail generation.
Google Drive and Dropbox scenarios use a local provider server and private endpoint overrides. They do
not contact real provider accounts. Chromium is the only configured browser project. Playwright traces
and screenshots are retained on failure under `test-results`.
Google Drive and Dropbox scenarios use a local provider server (`scripts/mock-provider-server.mjs`)
and never contact real provider accounts. The endpoint overrides that point at it are refused unless
`ALLOW_PROVIDER_ENDPOINT_OVERRIDES=true` **and** the override is a loopback URL - these endpoints
receive the OAuth client secret and refresh token, so they must not be redirectable in a deployed
environment. `npm run test:bdd` sets the flag; nothing else should.
The mock's recorded requests, refresh counter and fail-uploads switch are reset before every scenario
by an auto fixture in `tests/bdd/fixtures.ts`. That reset is also why the suite runs with a single
worker: the mock is one shared process, so parallel scenarios would reset each other's state. A global
teardown deletes the users each run creates, so repeated local runs do not need a database reset.
Chromium is the only configured browser project. Playwright traces and screenshots are retained on
failure under `test-results`.
Known gap: the password-reset scenarios use an ordinary signed-in session rather than a recovery link,
because following a real recovery link currently bounces to `/signin` - the browser client in
`src/routes/+layout.ts` has no cookie `set`/`remove` method, so it cannot persist the session it parses
out of the URL. `createRecoveryLink` in `tests/bdd/support/app.ts` is ready for when that is fixed.
The current National Dex maximum is deliberately asserted as 1025. When adding a new generation,
update that expectation together with Pokémon data, the corresponding game/dex files, database seed,