432 lines
14 KiB
Markdown
432 lines
14 KiB
Markdown
---
|
|
name: playwright-testing
|
|
description: Build, debug, and extend the current Playwright test setup for API, desktop E2E, mobile E2E, and visual checks. Use when changing tests, seeding deterministic content, or validating frontend/API behavior against the reverse-proxied CODING_URL.
|
|
---
|
|
|
|
# playwright-testing
|
|
|
|
## When to use this skill
|
|
|
|
Use this skill when:
|
|
|
|
- Adding or updating Playwright API, E2E, mobile, or visual tests
|
|
- Debugging failing tests in `tests/`
|
|
- Extending deterministic seed data for frontend or API coverage
|
|
- Verifying how `CODING_URL`, `ADMIN_TOKEN`, and collection permissions affect tests
|
|
- Deciding where a frontend assertion belongs: API spec, E2E spec, or visual regression spec
|
|
|
|
---
|
|
|
|
## Current test architecture
|
|
|
|
This starter uses Playwright across four slices:
|
|
|
|
- `tests/api/` for API-level checks
|
|
- `tests/e2e/` for desktop browser behavior
|
|
- `tests/e2e-admin/` for committed admin smoke coverage
|
|
- `tests/e2e-mobile/` for mobile behavior
|
|
- `tests/e2e-visual/` for screenshot-based regression tests
|
|
|
|
The current baseline is deterministic and seed-driven, not demo-content-driven.
|
|
|
|
### Core files
|
|
|
|
| File | Responsibility |
|
|
| ----------------------------------- | -------------------------------------------------------------------------------- |
|
|
| `playwright.config.ts` | Playwright projects, `baseURL`, global setup/teardown, BrowserSync-safe defaults |
|
|
| `tests/fixtures/test-constants.ts` | `ADMIN_TOKEN`, `API_BASE`, `TEST_BASE_URL`, seeded route constants |
|
|
| `tests/global-setup.ts` | Verifies `TEST_BASE_URL`, probes `/api`, seeds deterministic content |
|
|
| `tests/global-teardown.ts` | Cleans seeded content and disposes shared API contexts |
|
|
| `tests/api/helpers/admin-api.ts` | Shared admin CRUD helper using the static `Token:` header |
|
|
| `tests/api/helpers/seed-data.ts` | Seed definitions and seed cleanup for deterministic content pages |
|
|
| `tests/fixtures/console-monitor.ts` | Fails browser-based tests on unexpected page, console, or request errors |
|
|
| `tests/e2e/fixtures.ts` | Desktop browser fixtures and SPA helpers |
|
|
| `tests/e2e-admin/fixtures.ts` | Admin login helpers and admin smoke fixture setup |
|
|
| `tests/e2e-mobile/fixtures.ts` | Mobile browser fixtures and hamburger-menu helpers |
|
|
|
|
---
|
|
|
|
## Environment prerequisites
|
|
|
|
### Always use the configured `CODING_URL`
|
|
|
|
Playwright uses `TEST_BASE_URL` from `tests/fixtures/test-constants.ts`.
|
|
|
|
Resolution order:
|
|
|
|
1. `process.env.CODING_URL`
|
|
2. `CODING_URL` from `.env`
|
|
3. fallback `http://localhost:3000`
|
|
|
|
For this project, prefer the reverse-proxied `CODING_URL` from `.env` whenever it serves both:
|
|
|
|
- `/`
|
|
- `/api/...`
|
|
|
|
If `/api/...` returns HTML instead of JSON, the seeded setup is not usable and `globalSetup` should fail fast.
|
|
|
|
### Admin host and default credentials
|
|
|
|
Admin browser tests use `TEST_ADMIN_BASE_URL` from `tests/fixtures/test-constants.ts`.
|
|
|
|
Resolution order:
|
|
|
|
1. `process.env.CODING_TIBIADMIN_URL`
|
|
2. `CODING_TIBIADMIN_URL` from `.env`
|
|
3. fallback `http://localhost:3000`
|
|
|
|
The current smoke setup assumes the default dev login unless overridden via env vars:
|
|
|
|
- `ADMIN_UI_USERNAME` default: `admin`
|
|
- `ADMIN_UI_PASSWORD` default: `admin`
|
|
|
|
Keep this only for local/dev smoke coverage. Do not turn production credentials into committed test defaults.
|
|
|
|
### Static project token vs JWT user auth
|
|
|
|
This distinction matters for tests:
|
|
|
|
- `Token:` is the static project/admin token from `api/config.yml.env`
|
|
- `X-Auth-Token` is a JWT user token from a login flow
|
|
|
|
Collection permissions under `user:` do **not** grant access for static `Token:` requests.
|
|
If tests seed or mutate a collection through `ADMIN_TOKEN`, that collection must define explicit token permissions like:
|
|
|
|
```yaml
|
|
permissions:
|
|
"token:${ADMIN_TOKEN}":
|
|
methods:
|
|
get: true
|
|
post: true
|
|
put: true
|
|
delete: true
|
|
```
|
|
|
|
This is required for collections the seed helper writes to, such as `content`.
|
|
|
|
---
|
|
|
|
## BrowserSync navigation rule
|
|
|
|
BrowserSync keeps a WebSocket open permanently. Because of that:
|
|
|
|
- do **not** wait for `networkidle`
|
|
- do **not** rely on `load`
|
|
- use `domcontentloaded`
|
|
|
|
The shared fixtures already patch navigation helpers accordingly.
|
|
When writing new tests, keep using the project fixtures rather than raw Playwright `test`.
|
|
|
|
## Console watcher
|
|
|
|
Browser-based fixtures attach `attachConsoleMonitor(page)` from `tests/fixtures/console-monitor.ts`.
|
|
|
|
This monitor records and fails tests on unexpected:
|
|
|
|
- `pageerror`
|
|
- `console.error`
|
|
- failed network requests except explicitly ignored infrastructure noise
|
|
|
|
The intent is to catch real frontend/runtime regressions even when visible assertions still pass.
|
|
Do not silence app bugs by broadening ignored patterns unless the noise is clearly external infrastructure.
|
|
|
|
---
|
|
|
|
## Deterministic seed strategy
|
|
|
|
The current setup seeds content through the public collection API plus the static `Token:` header.
|
|
|
|
Use a hidden per-collection marker field as the default seed identity strategy.
|
|
In this project the convention is `_testdata: true`.
|
|
|
|
### Seed lifecycle
|
|
|
|
1. `globalSetup` probes the configured base URL.
|
|
2. `globalSetup` verifies `/api/content` returns JSON.
|
|
3. `globalSetup` removes old seeded entries by their hidden test marker before recreating them.
|
|
4. `globalSetup` creates deterministic seed entries.
|
|
5. Tests run against those seeded routes.
|
|
6. `globalTeardown` removes seeded entries again.
|
|
|
|
Setup cleanup and teardown cleanup are both required.
|
|
The setup cleanup handles leftovers from aborted or previously failed runs.
|
|
The teardown cleanup keeps the environment clean after successful runs.
|
|
|
|
### Hidden seed marker pattern
|
|
|
|
Prefer this pattern for every collection that may receive test-created data:
|
|
|
|
1. Add a hidden boolean field named `_testdata` as the last field in the collection schema.
|
|
2. Set `_testdata: true` on every seeded entry.
|
|
3. Let cleanup match `_testdata === true` first.
|
|
4. Keep older identifiers such as fixed paths or translation keys only as migration fallbacks when existing seed data already used them.
|
|
|
|
This is more robust than relying on translation keys because not every collection has a natural grouping field.
|
|
It also makes leftovers from aborted runs discoverable across heterogeneous collection shapes.
|
|
|
|
### Parallel worker rule
|
|
|
|
Seed creation and seed cleanup must remain run-scoped, not worker-scoped.
|
|
|
|
- perform seed cleanup and creation in `globalSetup`
|
|
- perform final seed cleanup in `globalTeardown`
|
|
- do not create or delete shared seeded data in per-test hooks or worker fixtures
|
|
- keep seeded identifiers deterministic so many workers can read the same seeded dataset safely
|
|
|
|
This project runs with many workers.
|
|
Parallel safety depends on one shared deterministic seed pass before the suite and one shared cleanup pass after the suite, not on each worker mutating shared fixtures independently.
|
|
|
|
### Current seeded routes
|
|
|
|
Defined in `tests/fixtures/test-constants.ts`:
|
|
|
|
- `SEEDED_TEST_CONTENT.home.path`
|
|
- `SEEDED_TEST_CONTENT.contact.path`
|
|
- `SEEDED_TEST_CONTENT.inactive.path`
|
|
|
|
These are backed by DE/EN content entries in `tests/api/helpers/seed-data.ts`.
|
|
|
|
### What the seed currently covers
|
|
|
|
- localized page routing
|
|
- hero rendering
|
|
- features rendering
|
|
- richtext rendering
|
|
- accordion rendering
|
|
- contact-form rendering
|
|
- inactive route -> 404 behavior
|
|
|
|
When adding new deterministic coverage, extend the seed data instead of asserting against editorial demo content.
|
|
|
|
---
|
|
|
|
## Which test type to use
|
|
|
|
### API tests
|
|
|
|
Use `tests/api/` when validating:
|
|
|
|
- collection filters
|
|
- public vs token-backed API behavior
|
|
- seeded content presence or absence
|
|
- mutation semantics independent of DOM rendering
|
|
|
|
Keep them narrow and data-oriented.
|
|
|
|
### Desktop E2E tests
|
|
|
|
Use `tests/e2e/` when validating:
|
|
|
|
- route changes
|
|
- language switching
|
|
- SPA navigation behavior
|
|
- block rendering in the real UI
|
|
- keyboard/a11y interactions such as skip links
|
|
|
|
### Admin smoke tests
|
|
|
|
Use `tests/e2e-admin/` when validating stable admin contracts such as:
|
|
|
|
- admin login still works in dev
|
|
- the project dashboard opens correctly
|
|
- core collections are still reachable
|
|
- critical collection views still render their configured labels/columns/actions
|
|
- collection lists render meaningful previews instead of broken placeholders
|
|
- important field widgets are configured and usable in entry forms
|
|
- pagebuilder block choosers, block forms, and live previews load correctly
|
|
|
|
These tests should stay intentionally narrow. They are regression guards for admin configuration, not full editor journey automation.
|
|
|
|
### Mobile E2E tests
|
|
|
|
Use `tests/e2e-mobile/` when validating:
|
|
|
|
- hamburger menu behavior
|
|
- responsive visibility
|
|
- mobile-specific navigation or controls
|
|
|
|
### Visual regression tests
|
|
|
|
Use `tests/e2e-visual/` only when layout/styling stability matters and a semantic DOM assertion is not enough.
|
|
|
|
## Admin config coverage strategy
|
|
|
|
Use a hybrid approach:
|
|
|
|
- committed Playwright smoke tests for stable, repeatable admin contracts
|
|
- one-shot MCP Playwright or VS Code browser checks for exploratory spot checks and ad-hoc audits
|
|
|
|
Committed tests should cover the admin paths that are expected to stay valid across everyday work, for example:
|
|
|
|
- login
|
|
- opening the Nova project dashboard
|
|
- visibility of the core collections
|
|
- opening important collection views like `content`
|
|
- checking that collection tables expose the intended columns, summaries, and preview thumbnails
|
|
- checking that key widgets like selects, foreign/media pickers, sidebars, and pagebuilder controls actually render
|
|
- checking that pagebuilder preview updates when block content changes
|
|
|
|
One-shot live browser checks are useful when:
|
|
|
|
- reviewing a newly added admin configuration once
|
|
- probing a flaky or hard-to-stabilize UI area before deciding what deserves a real test
|
|
- checking something highly visual or temporarily environment-specific
|
|
|
|
Do not rely on one-shot browser checks as the only safeguard for important admin paths. If a check matters repeatedly, promote it into `tests/e2e-admin/`.
|
|
|
|
---
|
|
|
|
## Current fixture conventions
|
|
|
|
### API
|
|
|
|
Use `tests/api/fixtures.ts`.
|
|
|
|
Current fixtures:
|
|
|
|
- `api`
|
|
- `adminApi`
|
|
|
|
`adminApi` is backed by the static `Token:` header from `ADMIN_TOKEN`.
|
|
|
|
### Desktop E2E
|
|
|
|
Use `tests/e2e/fixtures.ts`.
|
|
|
|
Helpers include:
|
|
|
|
- `waitForSpaReady(page)`
|
|
- `navigateToRoute(page, routePath)`
|
|
- `clickSpaLink(page, selector)`
|
|
- automatic console/page/request error monitoring via `attachConsoleMonitor(page)`
|
|
|
|
### Admin E2E
|
|
|
|
Use `tests/e2e-admin/fixtures.ts`.
|
|
|
|
Helpers include:
|
|
|
|
- `loginToAdmin(page)`
|
|
- `openNovaProjectDashboard(page)`
|
|
- automatic console/page/request error monitoring via `attachConsoleMonitor(page)`
|
|
|
|
### Mobile E2E
|
|
|
|
Use `tests/e2e-mobile/fixtures.ts`.
|
|
|
|
Helpers include:
|
|
|
|
- `waitForSpaReady(page)`
|
|
- `navigateToRoute(page, routePath)`
|
|
- `openHamburgerMenu(page)`
|
|
- `closeHamburgerMenuViaEscape(page)`
|
|
- automatic console/page/request error monitoring via `attachConsoleMonitor(page)`
|
|
|
|
### Visual E2E
|
|
|
|
Use `tests/e2e-visual/fixtures.ts`.
|
|
|
|
Helpers include:
|
|
|
|
- `waitForVisualReady(page)`
|
|
- `prepareForScreenshot(page)`
|
|
- `expectScreenshot(page, name, opts)`
|
|
- automatic console/page/request error monitoring via `attachConsoleMonitor(page)`
|
|
|
|
Do not reintroduce the old starter `authedPage` / `testUser` assumptions unless the project really needs JWT-user coverage again.
|
|
|
|
---
|
|
|
|
## Writing stable selectors
|
|
|
|
Prefer selectors that reflect current rendered structure and locale:
|
|
|
|
- scope language links to the relevant container (`header`, `main`, footer)
|
|
- avoid ambiguous `getByRole()` selectors when the same link text appears twice
|
|
- use the actual locale strings from `frontend/src/lib/i18n/locales/*.json`
|
|
- prefer stable block markers like `data-block="hero"`
|
|
|
|
Examples:
|
|
|
|
```ts
|
|
const header = page.locator("header")
|
|
await header.getByRole("link", { name: "en", exact: true }).click()
|
|
|
|
const homeLink = page.locator("main").getByRole("link", { name: "Zur Startseite" })
|
|
```
|
|
|
|
---
|
|
|
|
## Recommended commands
|
|
|
|
Run only the slice you changed.
|
|
|
|
```bash
|
|
/usr/bin/node ./node_modules/playwright/cli.js test tests/api/health.spec.ts --project=api
|
|
/usr/bin/node ./node_modules/playwright/cli.js test tests/e2e/home.spec.ts tests/e2e/demo.spec.ts --project=chromium
|
|
/usr/bin/node ./node_modules/playwright/cli.js test tests/e2e-admin/smoke.spec.ts --project=admin
|
|
/usr/bin/node ./node_modules/playwright/cli.js test tests/e2e-mobile/home.mobile.spec.ts --project=mobile-iphonese
|
|
```
|
|
|
|
If the shell environment is broken, calling the Playwright CLI through `/usr/bin/node` is acceptable in this workspace.
|
|
|
|
---
|
|
|
|
## Common failure patterns
|
|
|
|
### `/api/...` returns HTML
|
|
|
|
Cause:
|
|
|
|
- wrong `CODING_URL`
|
|
- fallback to BrowserSync without usable API proxy
|
|
|
|
Fix:
|
|
|
|
- verify the configured `CODING_URL` serves both `/` and `/api/...`
|
|
|
|
### `403 {"error":"empty token"}` on POST/PUT/DELETE
|
|
|
|
Cause:
|
|
|
|
- collection allows `user:` but not `"token:${ADMIN_TOKEN}":`
|
|
|
|
Fix:
|
|
|
|
- add explicit token permissions on that collection
|
|
|
|
### strict mode violations in role selectors
|
|
|
|
Cause:
|
|
|
|
- multiple matching links in header/footer/mobile menu
|
|
|
|
Fix:
|
|
|
|
- scope selectors to the intended container
|
|
- use `exact: true` where needed
|
|
|
|
### 404 assertions fail by link text
|
|
|
|
Cause:
|
|
|
|
- locale-specific text mismatch
|
|
|
|
Fix:
|
|
|
|
- use the real translated string from the locale JSON files
|
|
|
|
---
|
|
|
|
## Change workflow
|
|
|
|
When extending or fixing tests:
|
|
|
|
1. Start from the failing spec or the exact behavior to cover.
|
|
2. Check whether the needed content already exists in `seed-data.ts`.
|
|
3. Extend seed data only if the behavior is not already representable.
|
|
4. Run only the affected Playwright project/spec files.
|
|
5. Fix selectors or seed shape before widening scope.
|
|
|
|
Keep the test basis deterministic. Do not fall back to existing editorial demo content just because it is already present in the database.
|