From c058ec760f8f514fd95fd3ff8a86e9d05ec29b9e Mon Sep 17 00:00:00 2001 From: Sebastian Frank Date: Tue, 12 May 2026 20:42:15 +0000 Subject: [PATCH] =?UTF-8?q?=E2=9C=A8=20feat:=20add=20playwright-testing=20?= =?UTF-8?q?skill=20documentation=20and=20enhance=20AGENTS.md=20with=20test?= =?UTF-8?q?ing=20references?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .agents/skills/playwright-testing/SKILL.md | 331 +++++++++++++++++++++ AGENTS.md | 3 +- tests/AGENTS.md | 27 +- tests/fixtures/test-constants.ts | 8 +- 4 files changed, 360 insertions(+), 9 deletions(-) create mode 100644 .agents/skills/playwright-testing/SKILL.md diff --git a/.agents/skills/playwright-testing/SKILL.md b/.agents/skills/playwright-testing/SKILL.md new file mode 100644 index 0000000..3173569 --- /dev/null +++ b/.agents/skills/playwright-testing/SKILL.md @@ -0,0 +1,331 @@ +--- +name: playwright-testing +description: Build, debug, and extend the current Playwright test setup for API, desktop E2E, mobile E2E, and visual checks. Use when changing tests, seeding deterministic content, or validating frontend/API behavior against the reverse-proxied CODING_URL. +--- + +# playwright-testing + +## When to use this skill + +Use this skill when: + +- Adding or updating Playwright API, E2E, mobile, or visual tests +- Debugging failing tests in `tests/` +- Extending deterministic seed data for frontend or API coverage +- Verifying how `CODING_URL`, `ADMIN_TOKEN`, and collection permissions affect tests +- Deciding where a frontend assertion belongs: API spec, E2E spec, or visual regression spec + +--- + +## Current test architecture + +This starter uses Playwright across four slices: + +- `tests/api/` for API-level checks +- `tests/e2e/` for desktop browser behavior +- `tests/e2e-mobile/` for mobile behavior +- `tests/e2e-visual/` for screenshot-based regression tests + +The current baseline is deterministic and seed-driven, not demo-content-driven. + +### Core files + +| File | Responsibility | +| ----------------------------------- | -------------------------------------------------------------------------------- | +| `playwright.config.ts` | Playwright projects, `baseURL`, global setup/teardown, BrowserSync-safe defaults | +| `tests/fixtures/test-constants.ts` | `ADMIN_TOKEN`, `API_BASE`, `TEST_BASE_URL`, seeded route constants | +| `tests/global-setup.ts` | Verifies `TEST_BASE_URL`, probes `/api`, seeds deterministic content | +| `tests/global-teardown.ts` | Cleans seeded content and disposes shared API contexts | +| `tests/api/helpers/admin-api.ts` | Shared admin CRUD helper using the static `Token:` header | +| `tests/api/helpers/seed-data.ts` | Seed definitions and seed cleanup for deterministic content pages | +| `tests/fixtures/console-monitor.ts` | Fails browser-based tests on unexpected page, console, or request errors | +| `tests/e2e/fixtures.ts` | Desktop browser fixtures and SPA helpers | +| `tests/e2e-mobile/fixtures.ts` | Mobile browser fixtures and hamburger-menu helpers | + +--- + +## Environment prerequisites + +### Always use the configured `CODING_URL` + +Playwright uses `TEST_BASE_URL` from `tests/fixtures/test-constants.ts`. + +Resolution order: + +1. `process.env.CODING_URL` +2. `CODING_URL` from `.env` +3. fallback `http://localhost:3000` + +For this project, prefer the reverse-proxied `CODING_URL` from `.env` whenever it serves both: + +- `/` +- `/api/...` + +If `/api/...` returns HTML instead of JSON, the seeded setup is not usable and `globalSetup` should fail fast. + +### Static project token vs JWT user auth + +This distinction matters for tests: + +- `Token:` is the static project/admin token from `api/config.yml.env` +- `X-Auth-Token` is a JWT user token from a login flow + +Collection permissions under `user:` do **not** grant access for static `Token:` requests. +If tests seed or mutate a collection through `ADMIN_TOKEN`, that collection must define explicit token permissions like: + +```yaml +permissions: + "token:${ADMIN_TOKEN}": + methods: + get: true + post: true + put: true + delete: true +``` + +This is required for collections the seed helper writes to, such as `content`. + +--- + +## BrowserSync navigation rule + +BrowserSync keeps a WebSocket open permanently. Because of that: + +- do **not** wait for `networkidle` +- do **not** rely on `load` +- use `domcontentloaded` + +The shared fixtures already patch navigation helpers accordingly. +When writing new tests, keep using the project fixtures rather than raw Playwright `test`. + +## Console watcher + +Browser-based fixtures attach `attachConsoleMonitor(page)` from `tests/fixtures/console-monitor.ts`. + +This monitor records and fails tests on unexpected: + +- `pageerror` +- `console.error` +- failed network requests except explicitly ignored infrastructure noise + +The intent is to catch real frontend/runtime regressions even when visible assertions still pass. +Do not silence app bugs by broadening ignored patterns unless the noise is clearly external infrastructure. + +--- + +## Deterministic seed strategy + +The current setup seeds content through the public collection API plus the static `Token:` header. + +### Seed lifecycle + +1. `globalSetup` probes the configured base URL. +2. `globalSetup` verifies `/api/content` returns JSON. +3. `globalSetup` removes old seeded entries by `translationKey`. +4. `globalSetup` creates deterministic seed entries. +5. Tests run against those seeded routes. +6. `globalTeardown` removes seeded entries again. + +### Current seeded routes + +Defined in `tests/fixtures/test-constants.ts`: + +- `SEEDED_TEST_CONTENT.home.path` +- `SEEDED_TEST_CONTENT.contact.path` +- `SEEDED_TEST_CONTENT.inactive.path` + +These are backed by DE/EN content entries in `tests/api/helpers/seed-data.ts`. + +### What the seed currently covers + +- localized page routing +- hero rendering +- features rendering +- richtext rendering +- accordion rendering +- contact-form rendering +- inactive route -> 404 behavior + +When adding new deterministic coverage, extend the seed data instead of asserting against editorial demo content. + +--- + +## Which test type to use + +### API tests + +Use `tests/api/` when validating: + +- collection filters +- public vs token-backed API behavior +- seeded content presence or absence +- mutation semantics independent of DOM rendering + +Keep them narrow and data-oriented. + +### Desktop E2E tests + +Use `tests/e2e/` when validating: + +- route changes +- language switching +- SPA navigation behavior +- block rendering in the real UI +- keyboard/a11y interactions such as skip links + +### Mobile E2E tests + +Use `tests/e2e-mobile/` when validating: + +- hamburger menu behavior +- responsive visibility +- mobile-specific navigation or controls + +### Visual regression tests + +Use `tests/e2e-visual/` only when layout/styling stability matters and a semantic DOM assertion is not enough. + +--- + +## Current fixture conventions + +### API + +Use `tests/api/fixtures.ts`. + +Current fixtures: + +- `api` +- `adminApi` + +`adminApi` is backed by the static `Token:` header from `ADMIN_TOKEN`. + +### Desktop E2E + +Use `tests/e2e/fixtures.ts`. + +Helpers include: + +- `waitForSpaReady(page)` +- `navigateToRoute(page, routePath)` +- `clickSpaLink(page, selector)` +- automatic console/page/request error monitoring via `attachConsoleMonitor(page)` + +### Mobile E2E + +Use `tests/e2e-mobile/fixtures.ts`. + +Helpers include: + +- `waitForSpaReady(page)` +- `navigateToRoute(page, routePath)` +- `openHamburgerMenu(page)` +- `closeHamburgerMenuViaEscape(page)` +- automatic console/page/request error monitoring via `attachConsoleMonitor(page)` + +### Visual E2E + +Use `tests/e2e-visual/fixtures.ts`. + +Helpers include: + +- `waitForVisualReady(page)` +- `prepareForScreenshot(page)` +- `expectScreenshot(page, name, opts)` +- automatic console/page/request error monitoring via `attachConsoleMonitor(page)` + +Do not reintroduce the old starter `authedPage` / `testUser` assumptions unless the project really needs JWT-user coverage again. + +--- + +## Writing stable selectors + +Prefer selectors that reflect current rendered structure and locale: + +- scope language links to the relevant container (`header`, `main`, footer) +- avoid ambiguous `getByRole()` selectors when the same link text appears twice +- use the actual locale strings from `frontend/src/lib/i18n/locales/*.json` +- prefer stable block markers like `data-block="hero"` + +Examples: + +```ts +const header = page.locator("header") +await header.getByRole("link", { name: "en", exact: true }).click() + +const homeLink = page.locator("main").getByRole("link", { name: "Zur Startseite" }) +``` + +--- + +## Recommended commands + +Run only the slice you changed. + +```bash +/usr/bin/node ./node_modules/playwright/cli.js test tests/api/health.spec.ts --project=api +/usr/bin/node ./node_modules/playwright/cli.js test tests/e2e/home.spec.ts tests/e2e/demo.spec.ts --project=chromium +/usr/bin/node ./node_modules/playwright/cli.js test tests/e2e-mobile/home.mobile.spec.ts --project=mobile-iphonese +``` + +If the shell environment is broken, calling the Playwright CLI through `/usr/bin/node` is acceptable in this workspace. + +--- + +## Common failure patterns + +### `/api/...` returns HTML + +Cause: + +- wrong `CODING_URL` +- fallback to BrowserSync without usable API proxy + +Fix: + +- verify the configured `CODING_URL` serves both `/` and `/api/...` + +### `403 {"error":"empty token"}` on POST/PUT/DELETE + +Cause: + +- collection allows `user:` but not `"token:${ADMIN_TOKEN}":` + +Fix: + +- add explicit token permissions on that collection + +### strict mode violations in role selectors + +Cause: + +- multiple matching links in header/footer/mobile menu + +Fix: + +- scope selectors to the intended container +- use `exact: true` where needed + +### 404 assertions fail by link text + +Cause: + +- locale-specific text mismatch + +Fix: + +- use the real translated string from the locale JSON files + +--- + +## Change workflow + +When extending or fixing tests: + +1. Start from the failing spec or the exact behavior to cover. +2. Check whether the needed content already exists in `seed-data.ts`. +3. Extend seed data only if the behavior is not already representable. +4. Run only the affected Playwright project/spec files. +5. Fix selectors or seed shape before widening scope. + +Keep the test basis deterministic. Do not fall back to existing editorial demo content just because it is already present in the database. diff --git a/AGENTS.md b/AGENTS.md index 36e0a4b..c8f0760 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -178,6 +178,7 @@ These skills provide deep-dive documentation. Use them by phase instead of treat | --- | --- | | `frontend-architecture` | Routing, state management, Svelte 5 patterns, API layer, error handling | | `tibi-ssr-caching` | SSR rendering and cache invalidation | +| `playwright-testing` | Extending or debugging seeded Playwright API, desktop, mobile, or visual tests | ### Quickstart roadmap for a new website @@ -205,7 +206,7 @@ Use this order when building a project from scratch on this starter: Use `nova-ai-editor-features` only when there is a concrete editorial workflow for it. 8. Final verification. - Confirm public site, admin authoring, pagebuilder rendering, navigation/media resolution, forms/actions, and SSR all work, then run `yarn build`, `yarn build:server`, and `yarn validate`. + Use `playwright-testing` for deterministic API/E2E coverage, confirm public site, admin authoring, pagebuilder rendering, navigation/media resolution, forms/actions, and SSR all work, then run `yarn build`, `yarn build:server`, and `yarn validate`. For one-off tasks, use the phase tables above as the lookup index instead of maintaining a second skill matrix here. diff --git a/tests/AGENTS.md b/tests/AGENTS.md index d998d79..b1f0a79 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -2,6 +2,8 @@ Playwright tests for E2E, API, mobile, and visual regression. Config in `playwright.config.ts`. +For the full current workflow, prefer the `playwright-testing` skill. + ## Running tests - All tests: `yarn test` @@ -11,19 +13,36 @@ Playwright tests for E2E, API, mobile, and visual regression. Config in `playwri - Single file: `npx playwright test tests/e2e/filename.spec.ts` - Single test: `npx playwright test -g "test name"` - After code changes, run only affected spec files — not the full suite. +- Prefer the configured `CODING_URL` from `.env` whenever it serves both `/` and `/api/...`. +- The current test baseline is deterministic and seed-driven, not demo-content-driven. ## BrowserSync workaround BrowserSync keeps a WebSocket open permanently, preventing `networkidle` and `load` from resolving. All fixture files override `page.goto()` and `page.reload()` to use `waitUntil: "domcontentloaded"` by default. Always use `domcontentloaded` for navigation waits. +## Console watcher + +- Browser-based fixtures attach `tests/fixtures/console-monitor.ts` automatically. +- Unexpected `pageerror`, `console.error`, and failed network requests fail the test. +- Treat ignored patterns as infrastructure-only noise, not as a place to hide app bugs. + +## Seeded setup + +- `tests/global-setup.ts` verifies the configured base URL, probes `/api/content`, and seeds deterministic content pages. +- `tests/global-teardown.ts` removes seeded content again and disposes shared API contexts. +- Seed data lives in `tests/api/helpers/seed-data.ts`. +- Seeded route constants live in `tests/fixtures/test-constants.ts`. +- If tests write to a collection through `ADMIN_TOKEN`, that collection must define explicit permissions like `"token:${ADMIN_TOKEN}":`. + ## Fixtures & helpers -- `e2e/fixtures.ts` — Shared fixtures (`authedPage`, `testUser`) and helpers (`waitForSpaReady`, `navigateToRoute`, `clickSpaLink`). +- `e2e/fixtures.ts` — Shared desktop helpers (`waitForSpaReady`, `navigateToRoute`, `clickSpaLink`) with BrowserSync-safe navigation defaults. - `e2e-visual/fixtures.ts` — Visual test helpers (`waitForVisualReady`, `hideDynamicContent`, `prepareForScreenshot`, `expectScreenshot`, `getDynamicMasks`). - `e2e-mobile/fixtures.ts` — Mobile helpers (`openHamburgerMenu`, `isMobileViewport`, `isTabletViewport`, `isBelowLg`). -- `api/fixtures.ts` — API fixtures (`api`, `authedApi`, `accessToken`). -- `api/helpers/` — API test utilities (`test-user.ts`, `admin-api.ts`, `maildev.ts`). -- `fixtures/test-constants.ts` — Central constants (`TEST_USER`, `ADMIN_TOKEN`, `API_BASE`). +- `api/fixtures.ts` — API fixtures (`api`, `adminApi`). +- `api/helpers/` — API test utilities (`admin-api.ts`, `seed-data.ts`, `maildev.ts`). +- `fixtures/test-constants.ts` — Central constants (`ADMIN_TOKEN`, `API_BASE`, `TEST_BASE_URL`, `SEEDED_TEST_CONTENT`). +- `api/helpers/test-user.ts` is legacy starter scaffolding and should only be reused if the project really needs JWT-user coverage again. ## Visual regression diff --git a/tests/fixtures/test-constants.ts b/tests/fixtures/test-constants.ts index 536fb63..48a6a4d 100644 --- a/tests/fixtures/test-constants.ts +++ b/tests/fixtures/test-constants.ts @@ -4,10 +4,10 @@ import path from "node:path" /** * Zentrale Test-Konstanten für alle Tests. * - * Passe diese Werte an dein Projekt an: - * - TEST_USER: E-Mail/Passwort für den E2E-Test-User - * - ADMIN_TOKEN: Token aus api/config.yml.env - * - API_BASE: API-Pfad (Standard: /api) + * Aktuell relevant fuer das deterministische Playwright-Setup: + * - ADMIN_TOKEN: statischer Token aus api/config.yml.env fuer Seed/Cleanup + * - TEST_BASE_URL: bevorzugt CODING_URL aus env/.env + * - SEEDED_TEST_CONTENT: feste Routen fuer API- und E2E-Tests */ export const TEST_USER = {