From 06c836337f268d72612a4fd77ad2e5a13f40ae7f Mon Sep 17 00:00:00 2001 From: apairon Date: Fri, 16 Oct 2026 12:09:43 +0000 Subject: [PATCH] Neuaufbau: Inhalte bis auf Einstieg entfernt, Startseite/Sidebar/Regeln bereinigt --- AGENTS.md | 19 +- README.md | 6 +- astro.config.mjs | 27 - src/content/docs/adr/0000-vorlage.md | 12 - .../docs/adr/0001-starlight-entscheidung.md | 28 - src/content/docs/anwender/uebersicht.md | 14 - src/content/docs/index.mdx | 26 +- src/content/docs/nova/collection-config.md | 2297 ---------------- src/content/docs/nova/collection-export.md | 331 --- src/content/docs/nova/custom-export-script.md | 121 - src/content/docs/nova/konzept.md | 24 - .../nova/test-plans/02-navigation-routing.md | 35 - .../nova/test-plans/05-collection-view.md | 48 - .../test-plans/07-entry-editor-advanced.md | 61 - .../docs/nova/test-plans/11-foreign-keys.md | 58 - .../docs/nova/test-plans/16-ai-integration.md | 45 - src/content/docs/nova/test-plans/17-ai-tts.md | 28 - .../docs/nova/test-plans/25-file-upload.md | 47 - .../docs/nova/test-plans/27-media-grid.md | 44 - .../docs/nova/test-plans/34-i18n-fields.md | 38 - .../docs/nova/test-plans/35-entry-i18n.md | 58 - .../nova/test-plans/36-voice-input-fields.md | 35 - .../docs/nova/test-plans/37-pagebuilder.md | 39 - .../docs/nova/test-plans/38-foreign-file.md | 53 - .../nova/test-plans/39-preview-resolver.md | 28 - .../docs/nova/test-plans/40-actions.md | 25 - .../nova/test-plans/41-search-index-status.md | 28 - .../docs/nova/test-plans/42-impersonation.md | 51 - .../docs/nova/test-plans/43-backup-restore.md | 93 - .../docs/nova/test-plans/44-snapshots.md | 93 - src/content/docs/reference/index.md | 12 - src/content/docs/server/01-architecture.md | 261 -- src/content/docs/server/02-configuration.md | 622 ----- src/content/docs/server/03-api-endpoints.md | 2410 ----------------- src/content/docs/server/04-collections.md | 1749 ------------ src/content/docs/server/05-authentication.md | 606 ----- src/content/docs/server/06-hooks.md | 1721 ------------ src/content/docs/server/07-realtime.md | 190 -- .../docs/server/08-file-upload-images.md | 292 -- src/content/docs/server/09-llm-integration.md | 260 -- src/content/docs/server/10-audit.md | 277 -- src/content/docs/server/11-jobs.md | 154 -- src/content/docs/server/12-deployment.md | 152 -- src/content/docs/server/13-openapi-metrics.md | 151 -- src/content/docs/server/14-security.md | 636 ----- src/content/docs/server/15-troubleshooting.md | 232 -- src/content/docs/server/16-cookbook.md | 686 ----- .../docs/server/17-field-level-permissions.md | 550 ---- src/content/docs/server/18-orgs-teams.md | 720 ----- src/content/docs/server/19-actions.md | 315 --- .../docs/server/20-referenced-read-safety.md | 75 - src/content/docs/server/21-backup-restore.md | 534 ---- .../docs/server/22-personal-access-tokens.md | 295 -- src/content/docs/server/23-mcp.md | 156 -- .../docs/server/24-ephemeral-rest-tokens.md | 75 - src/content/docs/server/uebersicht.md | 21 - 56 files changed, 22 insertions(+), 16942 deletions(-) delete mode 100644 src/content/docs/adr/0000-vorlage.md delete mode 100644 src/content/docs/adr/0001-starlight-entscheidung.md delete mode 100644 src/content/docs/anwender/uebersicht.md delete mode 100644 src/content/docs/nova/collection-config.md delete mode 100644 src/content/docs/nova/collection-export.md delete mode 100644 src/content/docs/nova/custom-export-script.md delete mode 100644 src/content/docs/nova/konzept.md delete mode 100644 src/content/docs/nova/test-plans/02-navigation-routing.md delete mode 100644 src/content/docs/nova/test-plans/05-collection-view.md delete mode 100644 src/content/docs/nova/test-plans/07-entry-editor-advanced.md delete mode 100644 src/content/docs/nova/test-plans/11-foreign-keys.md delete mode 100644 src/content/docs/nova/test-plans/16-ai-integration.md delete mode 100644 src/content/docs/nova/test-plans/17-ai-tts.md delete mode 100644 src/content/docs/nova/test-plans/25-file-upload.md delete mode 100644 src/content/docs/nova/test-plans/27-media-grid.md delete mode 100644 src/content/docs/nova/test-plans/34-i18n-fields.md delete mode 100644 src/content/docs/nova/test-plans/35-entry-i18n.md delete mode 100644 src/content/docs/nova/test-plans/36-voice-input-fields.md delete mode 100644 src/content/docs/nova/test-plans/37-pagebuilder.md delete mode 100644 src/content/docs/nova/test-plans/38-foreign-file.md delete mode 100644 src/content/docs/nova/test-plans/39-preview-resolver.md delete mode 100644 src/content/docs/nova/test-plans/40-actions.md delete mode 100644 src/content/docs/nova/test-plans/41-search-index-status.md delete mode 100644 src/content/docs/nova/test-plans/42-impersonation.md delete mode 100644 src/content/docs/nova/test-plans/43-backup-restore.md delete mode 100644 src/content/docs/nova/test-plans/44-snapshots.md delete mode 100644 src/content/docs/reference/index.md delete mode 100644 src/content/docs/server/01-architecture.md delete mode 100644 src/content/docs/server/02-configuration.md delete mode 100644 src/content/docs/server/03-api-endpoints.md delete mode 100644 src/content/docs/server/04-collections.md delete mode 100644 src/content/docs/server/05-authentication.md delete mode 100644 src/content/docs/server/06-hooks.md delete mode 100644 src/content/docs/server/07-realtime.md delete mode 100644 src/content/docs/server/08-file-upload-images.md delete mode 100644 src/content/docs/server/09-llm-integration.md delete mode 100644 src/content/docs/server/10-audit.md delete mode 100644 src/content/docs/server/11-jobs.md delete mode 100644 src/content/docs/server/12-deployment.md delete mode 100644 src/content/docs/server/13-openapi-metrics.md delete mode 100644 src/content/docs/server/14-security.md delete mode 100644 src/content/docs/server/15-troubleshooting.md delete mode 100644 src/content/docs/server/16-cookbook.md delete mode 100644 src/content/docs/server/17-field-level-permissions.md delete mode 100644 src/content/docs/server/18-orgs-teams.md delete mode 100644 src/content/docs/server/19-actions.md delete mode 100644 src/content/docs/server/20-referenced-read-safety.md delete mode 100644 src/content/docs/server/21-backup-restore.md delete mode 100644 src/content/docs/server/22-personal-access-tokens.md delete mode 100644 src/content/docs/server/23-mcp.md delete mode 100644 src/content/docs/server/24-ephemeral-rest-tokens.md delete mode 100644 src/content/docs/server/uebersicht.md diff --git a/AGENTS.md b/AGENTS.md index fd3bd0f..282be97 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,21 +8,24 @@ Regeln für alle Agenten, die an dieser Doku oder an Tibi-Server / Tibi-Admin-No - **Definition of Done:** Jede PR an `tibi-server` oder `tibi-admin-nova`, die API, Hooks, Collections, Permissions, Jobs, Realtime, LLM, Deploy oder UX ändert, aktualisiert auch die Docs — sonst ist die PR nicht fertig. - **Sprache:** Deutsch, saubere Umlaute (ä ö ü ß), Du-Form für Anleitungen, kurze Sätze. -## 2. Wo was hingehört (Diátaxis) +## 2. Wo was hingehört (Diátaxis) — Stand: Neuaufbau + +Aktuell existiert nur `einstieg/`. Weitere Bereiche werden sauber neu aufgebaut; +pro Bereich Seite anlegen, Sidebar-Gruppe in `astro.config.mjs` ergänzen. | Ordner | Inhalt | Beispiel | |---|---|---| | `einstieg/` | Tutorials, Erste Schritte | Schnellstart, Begriffe | -| `server/` | **SYNC — nicht von Hand editieren**, Quelle `../tibi-server/docs/` | Collections, Hooks | -| `nova/` | Teils SYNC aus `../tibi-admin-nova/docs/`, plus `konzept.md` | Sammlungen, Export | -| `anwender/` | Redakteurs-Anleitungen (Aufgaben, keine Konzepte) | Anmelden, Pflegen | -| `reference/` | **Generiert — nie von Hand**, OpenAPI/Typen/Schema | openapi.md | -| `adr/` | Architektur-Entscheide, Format siehe `0000-vorlage.md` | 0001-starlight | +| `server/` (geplant) | **SYNC — nicht von Hand editieren**, Quelle `../tibi-server/docs/` | Collections, Hooks | +| `nova/` (geplant) | Teils SYNC aus `../tibi-admin-nova/docs/`, plus Konzeptseite | Sammlungen, Export | +| `anwender/` (geplant) | Redakteurs-Anleitungen (Aufgaben, keine Konzepte) | Anmelden, Pflegen | +| `reference/` (geplant) | **Generiert — nie von Hand**, OpenAPI/Typen/Schema | openapi.md | +| `adr/` (geplant) | Architektur-Entscheide, Format siehe Vorlage im Git-Verlauf (`adr/0000-vorlage.md`) | Starlight-Entscheid | ## 3. Neue Seite anlegen - Datei `src/content/docs//.md` mit Frontmatter `title` + `description`. -- Sidebar: `einstieg`/`anwender` explizit in `astro.config.mjs`, `server`/`nova`/`reference`/`adr` per `autogenerate` (kein Config-Eintrag nötig). +- Sidebar: jede Gruppe explizit in `astro.config.mjs` eintragen (beim Neuaufbau pro Bereich ergänzen; `autogenerate` erst nutzen, sobald der Ordner gepflegt befüllt ist). - Bilder nach `src/assets/`, max. 200 KB, deutsche Alt-Texte. ## 4. Sync-Workflow @@ -33,7 +36,7 @@ npm run sync npm run build # muss fehlerfrei durchlaufen ``` -- `server/uebersicht.md` und `nova/konzept.md` sind Handarbeit und werden vom Sync nie überschrieben (`keep`). +- `server/uebersicht.md` und `nova/konzept.md` sind Handarbeit und werden vom Sync nie überschrieben (`keep`) — beim Neuaufbau zuerst diese beiden Seiten neu anlegen, dann syncen. - Sync-Artefakte werden mitcommittet (statischer Build braucht sie). ## 5. Verbote diff --git a/README.md b/README.md index 4ae2fb4..3af35d7 100644 --- a/README.md +++ b/README.md @@ -33,11 +33,7 @@ Eigene Seiten: `src/content/docs/{einstieg,nova,anwender,reference,adr}/`. src/content/docs/ index.mdx Startseite (Hero) einstieg/ was-ist-tibi, schnellstart, begriffe - server/ SYNC aus tibi-server/docs (nicht von Hand editieren) - nova/ SYNC aus tibi-admin-nova/docs (+ konzept.md) - anwender/ Platzhalter Redakteurs-Doku - reference/ generiert (openapi), nicht von Hand - adr/ Architektur-Entscheide + # Neuaufbau: server/, nova/, anwender/, reference/, adr/ folgen scripts/ sync-server-docs.mjs sync-openapi.mjs diff --git a/astro.config.mjs b/astro.config.mjs index 1954bbf..2d29793 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -44,33 +44,6 @@ export default defineConfig({ { label: 'Begriffe', slug: 'einstieg/begriffe' }, ], }, - { - label: 'Tibi-Server', - collapsed: true, - items: [{ autogenerate: { directory: 'server' } }], - }, - { - label: 'Tibi-Admin-Nova', - collapsed: true, - items: [{ autogenerate: { directory: 'nova' } }], - }, - { - label: 'Für Anwender', - collapsed: true, - items: [ - { label: 'Übersicht (bald)', slug: 'anwender/uebersicht' }, - ], - }, - { - label: 'Referenz', - collapsed: true, - items: [{ autogenerate: { directory: 'reference' } }], - }, - { - label: 'Architektur-Entscheide (ADR)', - collapsed: true, - items: [{ autogenerate: { directory: 'adr' } }], - }, ], head: [ { diff --git a/src/content/docs/adr/0000-vorlage.md b/src/content/docs/adr/0000-vorlage.md deleted file mode 100644 index 02b508a..0000000 --- a/src/content/docs/adr/0000-vorlage.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: 'ADR-0000: Vorlage' -description: Vorlage für Architektur-Entscheidungen. ---- - -# ADR-0000: Titel - -- **Status:** vorgeschlagen -- **Datum:** 2026-10-16 -- **Kontext:** Warum musste entschieden werden? -- **Entscheidung:** Was wurde beschlossen? -- **Konsequenzen:** Was folgt daraus (gut/schlecht)? diff --git a/src/content/docs/adr/0001-starlight-entscheidung.md b/src/content/docs/adr/0001-starlight-entscheidung.md deleted file mode 100644 index 2e6e551..0000000 --- a/src/content/docs/adr/0001-starlight-entscheidung.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 'ADR-0001: Starlight statt Svelte-Starter' -description: Warum die Doku auf Astro Starlight basiert. ---- - -# ADR-0001: Starlight statt Svelte-Starter als Doku-Basis - -- **Status:** beschlossen -- **Datum:** 2026-10-16 - -## Kontext - -Eigene Doku-Seite für Entwickler (später Anwender) zu Tibi-Server und Tibi-Admin-Nova. Anforderungen: Markdown-Quellen, gerenderte durchsuchbare Webseite, statisch hostbar, agentenpflegbar. `tibi-svelte-starter` stand als Basis im Raum. - -## Entscheidung - -Eigenes Repo `cms/tibi-docs` auf **Astro Starlight**-Basis, statischer Build, Pagefind-Suche, Traefik-Docker-Stack wie üblich, Theme an Nova angelehnt. - -## Begründung - -- Starter/Nova-Stack (SPA + SSR-Hook + Admin-Bundle + Docker-Verbund) ist für Doku Overkill; Suche/Sidebar/TOC/Versionierung müssten selbst gebaut werden. -- Starlight liefert das alles gratis, Markdown bleibt Single Source, Build ist ohne Backend deploybar. -- Altes `tibi-docs` (Demo+Docs vermischt) wird zu `tibi-docs-legacy` archiviert. - -## Konsequenzen - -- Doku-Repo braucht Node/Astro-Know-how statt nur Svelte/esbuild. -- Server-Docs werden per `npm run sync` übernommen, nicht geforkt. diff --git a/src/content/docs/anwender/uebersicht.md b/src/content/docs/anwender/uebersicht.md deleted file mode 100644 index aec3937..0000000 --- a/src/content/docs/anwender/uebersicht.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Übersicht für Anwender -description: Platzhalter für die spätere Redakteurs-Dokumentation. ---- - -> **Noch im Aufbau.** Hier entsteht die Anwender-Doku für Redakteure: Anmelden, Inhalte pflegen, Medien hochladen, Seiten bauen, freigeben. - -Geplant: - -- Anmelden und Rollen -- Sammlungen und Dokumente anlegen/bearbeiten -- Medien und Bilder -- Exporte -- Häufige Fragen diff --git a/src/content/docs/index.mdx b/src/content/docs/index.mdx index 6618143..3428abb 100644 --- a/src/content/docs/index.mdx +++ b/src/content/docs/index.mdx @@ -8,31 +8,25 @@ hero: - text: Schnellstart link: /einstieg/schnellstart/ icon: right-arrow - - text: Tibi-Server - link: /server/uebersicht/ - variant: minimal - - text: Tibi-Admin-Nova - link: /nova/konzept/ + - text: Was ist Tibi? + link: /einstieg/was-ist-tibi/ variant: minimal --- import { Card, CardGrid } from '@astrojs/starlight/components'; -![So sieht Tibi-Admin-Nova aus: Eintrag pflegen mit Toolbar, Formular und Rich-Text-Editor.](../../assets/admin-nova.png) - ## Wo finde ich was? +> **Im Aufbau.** Diese Doku wird gerade sauber neu strukturiert. Fertig ist der [Einstieg](/einstieg/was-ist-tibi/) — Server-, Nova-, Anwender- und Referenz-Seiten folgen Schritt für Schritt. + - - Was ist Tibi, Schnellstart mit Docker, Begriffe von Namespace bis Collection. [Los geht's](/einstieg/was-ist-tibi/). + + Überblick über Tibi-Server, Tibi-Admin-Nova und Tibi-Projekte. [Lesen](/einstieg/was-ist-tibi/). - - Go-Backend: Collections-YAML, Hooks, Auth, Jobs, Realtime, LLM, Deploy. [Zur Übersicht](/server/uebersicht/). + + Doku und Stacks lokal starten, Doku pflegen. [Starten](/einstieg/schnellstart/). - - Svelte-Admin: Sammlungen pflegen, Exporte, Rollen, Testpläne. [Zum Konzept](/nova/konzept/). - - - Autogeneriert: OpenAPI-Spec, Hook-JS-API, Collection-Schema. [Zur Referenz](/reference/). + + Namespace, Collection, Hook und Co. im Glossar. [Nachschlagen](/einstieg/begriffe/). diff --git a/src/content/docs/nova/collection-config.md b/src/content/docs/nova/collection-config.md deleted file mode 100644 index b3f2409..0000000 --- a/src/content/docs/nova/collection-config.md +++ /dev/null @@ -1,2297 +0,0 @@ ---- -title: 'Collection-Config Referenz' ---- - -# Collection-Config Referenz - -Vollständige Dokumentation aller Konfigurationsmöglichkeiten für tibi-server Collections. -Gilt für **tibi-admin-nova**, **tibi-admin (Legacy)** und den **tibi-server**. - -> **Stand:** Februar 2026 -> **Quellen:** `tibi-types/schemas/api-config/`, `types/admin.d.ts`, `for_reference/tibi-server/` - ---- - -## Inhaltsverzeichnis - -1. [Projekt-Konfiguration (config.yml)](#1-projekt-konfiguration-configyml) -2. [Collection-Schema](#2-collection-schema) -3. [Collection Meta](#3-collection-meta) -4. [Feld-Schema](#4-feld-schema) -5. [Feld-Typen (Server)](#5-feld-typen-server) -6. [Widget-Typen (Admin UI)](#6-widget-typen-admin-ui) -7. [Feld-Meta](#7-feld-meta) -8. [Choices-Konfiguration](#8-choices-konfiguration) -9. [Foreign-Key-Konfiguration](#9-foreign-key-konfiguration) -10. [Validator](#10-validator) -11. [Search (Collection-Level)](#11-search-collection-level) -12. [Views (List-Ansichten)](#12-views-list-ansichten) -13. [Kompatibilitätsmatrix Legacy → Nova](#13-kompatibilitätsmatrix-legacy--nova) -14. [Bekannte Inkompatibilitäten & TODOs](#14-bekannte-inkompatibilitäten--todos) - ---- - -## 1. Projekt-Konfiguration (config.yml) - -Die Hauptkonfiguration liegt in `api/config.yml`. Sie definiert Namespace, Collections und globale Einstellungen. - -**Schema:** `tibi-types/schemas/api-config/config.json` - -```yaml -namespace: my_project # MongoDB-Datenbank-Suffix & API-URL-Teil (required) - -meta: - dashboard: path/to/dashboard.yml # Dashboard-Konfiguration - imageUrl: "https://..." # Projekt-Bild für Admin-UI - # oder dynamisch: - imageUrl: - eval: "`https://example.com/${context.namespace}/logo.png`" - permissions: # Berechtigungsgruppen-Labels - - name: pages - label: - de: Seiten - en: Pages - -collections: - - !include collections/pages.yml # Externe YAML-Dateien - - !include collections/blog.yml - - name: inline_collection # Oder inline definiert - permissions: - get: public - fields: [...] - -jobs: - - !include jobs/cleanup.yml # Geplante Aufgaben - -assets: - - name: dist # Statische Asset-Verzeichnisse - path: ./dist - -cors: {} -``` - -## 2. Collection-Schema - -Jede Collection definiert Name, Berechtigungen, Felder und optionale Metadaten. - -**Schema:** `tibi-types/schemas/api-config/collection.json` - -| Feld | Typ | Required | Beschreibung | -|------|-----|----------|-------------| -| `name` | `string` | ✅ | Collection-Name, Teil der API-URL (`/api/{namespace}/{name}`) | -| `permissions` | `object` | ✅ | Zugriffsregeln (siehe unten) | -| `fields` | `Field[]` | | Felddefinitionen (siehe [Feld-Schema](#4-feld-schema)) | -| `meta` | `object` | | Admin-UI-Konfiguration (siehe [Collection Meta](#3-collection-meta)) | -| `defaultLanguage` | `string` | | Sprache für MongoDB Text-Index (z.B. `"german"`) | -| `uploadPath` | `string` | | Pfad für Datei-Uploads (relativ zu config.yml) | -| `upload` | `object` | | Upload-Limits und erlaubte MIME-Typen | -| `hooks` | `object` | | JS-Hooks für CRUD-Operationen | -| `projections` | `object` | | Benannte MongoDB-Projektionen | -| `imageFilter` | `object` | | Bild-Resize-Konfigurationen | -| `indexes` | `array` | | Zusätzliche MongoDB-Indizes | -| `audit` | `object` | | Audit-Log-Konfiguration für CRUD/Bulk-Aktionen | -| `bulk` | `object` | | Bulk-Optimierungen für Create/Update/Delete | -| `queryLimits` | `object` | | Default-/Max-Limits für Listen-Queries | -| `readonlyFields` | `string[]` | | Globale serverseitige Readonly-Felder | -| `hiddenFields` | `string[]` | | Globale serverseitige Hidden-Felder | -| `strictFields` | `boolean` | | Unbekannte Felder strikt ablehnen | -| `search` | `SearchConfig[]` | | Benannte Suchmodi für die Nova-Suche | - -### Permissions - -Es gibt zwei gebräuchliche Formen: - -1. Kurzform pro HTTP-Methode -2. Ausführliches Permission-Objekt mit `public`, `user`, eigenen Rollen oder `token:*` - -```yaml -# Kurzform: -permissions: - get: public # Alle dürfen lesen - post: "token:editor" # Nur Token mit 'editor' Berechtigung - put: user # Jeder eingeloggte User - delete: "token:admin" # Nur Admin-Token - -# Vollform mit PermissionSets: -permissions: - public: - methods: - get: true - user: - methods: - get: true - post: true - put: true - readonlyFields: - - seo.slug - hiddenFields: - - internalNotes - token:admin: - methods: - get: true - post: true - put: true - delete: true - validProjections: [navigation, teaser] - editors: - methods: - get: true - post: true - put: true - filter: - status: draft -``` - -Wichtige Felder im PermissionSet: - -- `methods`: Rechte pro HTTP-Methode -- `filter`: serverseitiger Filter auf alle Queries dieser Rolle -- `validProjections`: erlaubte Projektionen für diese Rolle -- `readonlyFields`: serverseitig schreibgeschützte Feldpfade -- `hiddenFields`: serverseitig verborgene Feldpfade - -Mögliche Rollen-Schlüssel: `public`, `user`, freie Namen wie `editors` und `token:{permission_name}`. - -### Projections - -```yaml -projections: - navigation: # Benannte Projektion - title: 1 - slug: 1 - sort: 1 - teaser: - title: 1 - cover: 1 - excerpt: 1 -``` - -### Image Filter - -```yaml -imageFilter: - xs: - fit: cover - width: 100 - height: 100 - quality: 80 - s: - fit: contain - width: 400 - height: 300 - m: - width: 800 - quality: 85 - resampling: lanczos - l: - width: 1200 - xl: - width: 1920 - quality: 90 -``` - -### Upload - -Collection-spezifische Upload-Beschränkungen. - -```yaml -upload: - maxBodySize: 25MB - allowedMimeTypes: - - image/jpeg - - image/png - - image/webp -``` - -### Hooks - -```yaml -hooks: - get: - read: hooks/my_hook.js # Vor dem Lesen - post: - bind: hooks/bind.js # Nach dem Binden der Daten - validate: hooks/validate.js # Validierung - create: hooks/create.js # Nach dem Erstellen - return: hooks/return.js # Vor der Rückgabe - put: - validate: hooks/validate.js - delete: - read: hooks/before_delete.js -``` - -### Indexes - -```yaml -indexes: - - keys: - slug: 1 - options: - unique: true - - keys: - title: "text" - content: "text" - options: - weights: - title: 10 - content: 1 -``` - -### Audit - -Audit-Logging für CRUD- und Bulk-Aktionen. - -```yaml -audit: - enabled: true - actions: [create, update, delete, bulkCreate, bulkUpdate, bulkDelete] -``` - -### Bulk - -Steuert serverseitige Bulk-Optimierungen. - -```yaml -bulk: - optimize: - create: true - update: true - delete: true -``` - -### Query Limits - -Begrenzt Default- und Maximal-Listenabfragen. - -```yaml -queryLimits: - defaultLimit: 50 - maxLimit: 500 -``` - -### readonlyFields / hiddenFields - -Globale Feldrestriktionen direkt an der Collection. Diese gelten serverseitig zusätzlich zu rollenspezifischen PermissionSets. - -```yaml -readonlyFields: - - slug - - seo.canonicalUrl - -hiddenFields: - - internalNotes - - billing.secret -``` - -### strictFields - -Lehnt unbekannte Felder beim Schreiben strikt ab. - -```yaml -strictFields: true -``` - -### Vollständiges Beispiel - -```yaml -name: pages -defaultLanguage: german -uploadPath: ./uploads/pages - -permissions: - get: public - post: user - put: user - delete: "token:admin" - -projections: - navigation: - title: 1 - slug: 1 - -imageFilter: - thumb: - fit: cover - width: 200 - height: 200 - quality: 80 - -hooks: - post: - create: hooks/clear_cache.js - put: - create: hooks/clear_cache.js - -meta: - label: - de: Seiten - en: Pages - muiIcon: file-document-outline - -fields: - - name: title - type: string - validator: - required: true - meta: - label: - de: Titel - en: Title - - name: slug - type: string - index: [unique] - meta: - label: Slug -``` - ---- - -## 3. Collection Meta - -Steuert die Darstellung und Navigation im Admin-UI. - -**Schema:** `tibi-types/schemas/api-config/collectionNavigation.json` -**Types:** `types/admin.d.ts → AdminCollectionMeta` - -### Übersicht aller Meta-Optionen - -| Option | Typ | Nova | Legacy | Beschreibung | -|--------|-----|------|--------|-------------| -| [`label`](#label) | `string \| I18nText` | ✅ | ✅ | Anzeigename in Navigation | -| [`muiIcon`](#muiicon) | `string` | ✅ | ✅ | Icon-Name (MDI) | -| [`singleton`](#singleton) | `{ enabled: boolean, key?: string }` | ✅ | ✅ | Nur ein Dokument / optionaler Singleton-Key | -| [`hide`](#hide) | `boolean` | ✅ | ✅ | Collection verstecken (Admin sieht Badge) | -| [`subNavigation`](#subnavigation) | `SubNav[]` | ✅ | ✅ | Gefilterte Unteransichten | -| [`group`](#group) | `string` | ✅ | ✅ | Navigationsgruppe | -| [`viewHint`](#viewhint) | `"table" \| "cards" \| "media" \| { navigation }` | ✅ | ❌ | Standard-/Spezialansicht der Collection | -| [`preview`](#preview) | `string \| { eval } \| PreviewSlots` | ✅ | ✅ | Entry-Vorschautext / strukturierte Preview | -| [`defaultSort`](#defaultsort) | `{ field, order }` | ✅ | ✅ | Standard-Sortierung | -| [`defaultCallback`](#defaultcallback) | `"edit" \| "view" \| { eval }` | ⚠️ | ✅ | Klick-Aktion | -| [`defaultImageFilter`](#defaultimagefilter) | `string` | ⚠️ | ✅ | Standard-Bildfilter | -| [`filter`](#filter) | `object` | ⚠️ | ✅ | Pre-Filter | -| [`views`](#views-meta) | `View[]` | ⚠️ | ✅ | List-View-Definitionen | -| [`i18n`](#i18n) | `I18nConfig \| false` | ✅ | ❌ | Collection-spezifische i18n-Konfiguration | -| [`sidebar`](#sidebar) | `SidebarGroup[]` | ✅ | ❌ | Geordnete Sidebar-Gruppen | -| [`sidebarGroups`](#sidebargroups-deprecated) | `Record` | 🗑️ | ✅ | Legacy-Sidebar-Gruppen | -| [`sections`](#sections) | `SectionConfig[]` | ✅ | ❌ | Benannte Sections (neues Feld-Format) | -| [`pagebuilder`](#pagebuilder) | `object` | ✅ | ❌ | Collection-Defaults für Pagebuilder | -| [`multiupload`](#multiupload) | `object` | ❌ | ✅ | Multi-Upload | -| [`navigation`](#navigation) | `object` | ❌ | ✅ | Custom Navigation | -| [`backup`](#backup) | `object` | ❌ | ✅ | Entry-Versionierung | -| [`multilingual`](#multilingual) | `object` | ❌ | ✅ | Mehrsprachigkeit | -| [`informationGuide`](#informationguide) | `object` | ❌ | ✅ | Info-Anzeige im Editor | -| [`rowIdentTpl`](#rowidenttpl) | `string \| { twig } \| { eval }` | 🗑️ | ✅ | → `preview` | -| [`tablist`](#tablist) | `object` | 🗑️ | ✅ | → `section` auf Feldern | -| [`quickEdit`](#quickedit) | `object` | 🗑️ | ✅ | → Nova Bulk-Edit | -| [`noDB`](#nodb) | `object` | 🗑️ | ✅ | Nischen-Feature | - -**Legende:** ✅ = Implementiert, ⚠️ = Teilweise/TODO, ❌ = Nicht implementiert, 🗑️ = Deprecated - ---- - -### label - -Anzeigename der Collection in der Sidebar-Navigation und Breadcrumbs. - -```yaml -meta: - label: Seiten # Einfacher String - # oder mehrsprachig: - label: - de: Seiten - en: Pages -``` - -### muiIcon - -Material Design Icon-Name für die Navigation. Verwendet [MDI Icons](https://pictogrammers.com/library/mdi/). - -```yaml -meta: - muiIcon: file-document-outline -``` - -### singleton - -Collection hat genau ein Dokument. Die Listenansicht wird übersprungen — der Eintrag öffnet sich direkt. - -```yaml -meta: - singleton: - enabled: true - - # Optional: mehrere Singletons in einer Collection unterscheiden - singleton: - enabled: true - key: site_settings -``` - -### hide - -Versteckt die Collection für Nicht-Admin-Benutzer. Admins sehen sie weiterhin mit einem lila "Versteckt"-Badge (analog zu versteckten Feldern). - -```yaml -meta: - hide: true -``` - -### subNavigation - -Erstellt gefilterte Unteransichten in der Navigation. Jeder Eintrag erscheint als separater Navigationspunkt. - -```yaml -meta: - subNavigation: - - name: all - label: - de: Alle - en: All - muiIcon: format-list-bulleted - - name: published - label: - de: Veröffentlicht - en: Published - muiIcon: check-circle - filter: - status: published - - name: drafts - label: - de: Entwürfe - en: Drafts - muiIcon: pencil - filter: - status: draft - defaultSort: - field: updateTime - order: DESC - setDefault: - field: status - value: draft -``` - -Zusätzlich unterstützt Nova pro Sub-Navigation: - -- `defaultSort`: überschreibt die Sortierung innerhalb dieses Tabs -- `setDefault`: vorbelegen eines Feldes beim Erstellen eines neuen Eintrags aus diesem Tab - -### group - -Gruppiert Collections in der Sidebar unter einem gemeinsamen Header. - -```yaml -meta: - group: content # Alle Collections mit group: content werden zusammengefasst -``` - -### viewHint - -Steuert die Standardansicht der Collection in Nova. - -```yaml -meta: - viewHint: table - # oder: - viewHint: cards - # oder: - viewHint: - media: - ai: - targetField: alt - prompt: Beschreibe das Bild kurz und sachlich für einen Alt-Text. - image: - maxWidth: 1280 - maxHeight: 1280 - quality: 0.82 -``` - -Semantik von `media`: - -- `upload.autoFill`: Mapping von Entry-Feldern auf Auto-Fill-Quellen beim Upload. -- `ai.targetField`: Dot-Notation des Felds, das nach Bildauswahl automatisch per KI befüllt werden soll. Unterstützt auch i18n-Objekte wie `localizedCaption` oder einzelne Sprachpfade wie `localizedCaption.de`. -- `ai.prompt`: optionale Anweisung für die Textart. Überschreibt den globalen Default aus `frontend/assets/system.config.json` unter `ai.vision.defaultPrompt`. -- `ai.image`: optionale Downscale-Konfiguration nur für den KI-Request. Überschreibt `ai.vision.image` aus der System-Config. Das beeinflusst nicht den normalen Upload-Downscale des File-Felds. - -Beispiel für `upload.autoFill`: - -```yaml -meta: - viewHint: - media: - upload: - autoFill: - title: name - alt: basename -``` - -Für i18n-Zielfelder ergänzt Nova automatisch das gewünschte Ausgabeformat. Die `prompt`-Config beschreibt also nur den gewünschten Texttyp, nicht das JSON-Format. Bereits ausgefüllte Zielfelder werden nicht überschrieben. - -Für rekursive Navigationen kann `viewHint` auch als Objekt verwendet werden. Nova erkennt die Collection dann als Navigation-Collection und rendert deklarierte Baum-Slots plus bestehende Baum-Dokumente. - -```yaml -meta: - viewHint: - navigation: - nodesField: nodes - declaredTrees: - - label: - de: Hauptnavigation DE - singleton: - treeName: main - lang: de - maxLevel: 3 - - label: - de: Service Navigation DE - singleton: - treeName: service - lang: de - maxLevel: 1 -``` - -Semantik von `navigation`: - -- `nodesField`: Root-Feld mit dem rekursiven `object[]`-Baum. Standard ist `nodes`. Dieses Feld bekommt in Nova automatisch den spezialisierten Tree-Editor. -- `declaredTrees`: Optionale deklarierte Singleton-Slots für Soll-Bäume. -- `singleton`: Root-Feldwerte, über die ein deklarierter Baum identifiziert wird. Fehlt das Dokument, zeigt Nova trotzdem einen Platzhalter und legt beim Öffnen einen neuen Baum mit diesen Werten plus leerem `nodesField` an. -- Nicht deklarierte, aber vorhandene Baum-Dokumente bleiben in der Liste sichtbar. - -> ⚠️ **Nova-Status:** Collection-Erkennung, Slot-Übersicht und ein erster rekursiver Tree-Editor für das `nodesField` sind implementiert. Tiefere spezialisierte Features wie echte Baum-Umhängungen zwischen Ebenen können später ergänzt werden. - -### preview - -Definiert, wie ein Entry textlich identifiziert wird — verwendet in Breadcrumbs, ForeignKey-Anzeige, Listenansichten. - -```yaml -meta: - # Einfach: Wert eines Feldes - preview: title - - # Dynamisch: JS-Ausdruck mit $this (Entry-Objekt) - preview: - eval: "`${$this.firstName} ${$this.lastName}`" - - # Strukturierte Preview-Slots: - preview: - label: title - secondary: subtitle - badge: status - image: cover - imageFallback: teaserImage - - # Explizite Table-Spalten für Nova-Listen: - preview: - label: title - table: - - title - - source: status - label: Status - - source: author.name - label: Autor - select: [author.name] -``` - -Nova unterstützt hier neben String/Eval auch strukturierte Preview-Slots (`label`, `secondary`, `tertiary`, `badge`, `image`, `imageFallback`, `mediaFile`) sowie explizite `table`-Spalten und `select` für `_select`-Optimierung. - -Alle Slots — inklusive `image`/`imageFallback` — akzeptieren alternativ `{ eval: "..." }`: -der Eval-Kontext entspricht dem Root-`preview.eval` (`$this`/`$` = Entry). Rückgabe kann -ein FileField-Objekt (z.B. `$this.cover`) oder eine Bild-URL sein. Bei Eval-Slots müssen -referenzierte Felder via `select` in `_select` aufgenommen werden, wenn die Liste nicht -ohnehin Full-Records lädt. - -### defaultSort - -Standard-Sortierung für die Listenansicht. - -```yaml -meta: - defaultSort: - field: title # Feldname - order: ASC # ASC | DESC | MANUALLY - - # Feld als Source-Objekt: - defaultSort: - field: - source: sort - order: MANUALLY # Drag-to-reorder -``` - -> ⚠️ **Nova-Status:** Basis-Sortierung (ASC/DESC) funktioniert. `MANUALLY` (Drag-Sortierung) noch nicht implementiert. - -> ⚠️ **Achtung:** `defaultSort` muss ein Objekt `{ field, order }` sein (siehe Schema). String-Format wie `"-publishDate"` wird NICHT vom Schema unterstützt — manche Legacy-Configs nutzen aber die kurzform `"-fieldName"` (Minus für DESC), die vom alten Admin intern konvertiert wurde. - -### defaultCallback - -Aktion beim Klick auf einen Eintrag in der Liste. - -```yaml -meta: - # Standard (default): - defaultCallback: edit - - # Nur-Lesen: - defaultCallback: view - - # Custom (z.B. selectEntry für ForeignKey-Modals): - defaultCallback: - eval: "parent.selectEntry($this)" -``` - -> ⚠️ **Nova-Status:** Immer "edit". View und eval noch nicht implementiert. - -### defaultImageFilter - -Standard-Bildfilter-Name für Thumbnails in Listenansichten. - -```yaml -meta: - defaultImageFilter: thumb # Referenziert imageFilter.thumb -``` - -### filter - -Pre-Filter auf Collection-Ebene. Einträge, die nicht matchen, werden in der Liste ausgeblendet. - -```yaml -meta: - filter: - type: page - active: true -``` - -> ⚠️ **Nova-Status:** Nicht implementiert (subNavigation-Filter funktionieren). - -### views (Meta) {#views-meta} - -Definiert Listenansichten. Siehe [Views (List-Ansichten)](#11-views-list-ansichten) für Details. - -> ⚠️ **Nova-Status:** Nova hat eine eingebaute responsive Tabelle + Cards-Toggle. Custom Views noch nicht unterstützt. - -### multiupload - -Drag-and-Drop von mehreren Dateien erstellt jeweils separate Einträge. - -```yaml -meta: - multiupload: - fields: - - source: file - prefilledFields: - - source: title - defaultValue: - eval: "$file.name.replace(/\\.[^.]+$/, '')" -``` - -> ❌ **Nova-Status:** Nicht implementiert. - -### navigation - -Custom Navigation-Elemente für die Collection. - -```yaml -meta: - navigation: - elements: - - name: external - label: Externe Seite - href: "https://example.com" - muiIcon: open-in-new - displayIfInActiveTab: true -``` - -> ❌ **Nova-Status:** Nicht implementiert. - -### backup - -Aktiviert Entry-Versionierung in einer separaten Backup-Collection. - -```yaml -meta: - backup: - active: true - collectionName: backups # Ziel-Collection für Versionen -``` - -> ❌ **Nova-Status:** Nicht implementiert. - -### multilingual - -Konfiguriert mehrsprachige Bearbeitung. - -```yaml -meta: - multilingual: - enabled: true - defaultLang: de - pathToLanguageSelect: lang - pathToLanguageReferences: translations -``` - -> ❌ **Nova-Status:** Nicht implementiert. - -### informationGuide - -Zeigt eine Informationsbox im Entry-Editor. - -```yaml -meta: - informationGuide: - title: - de: Hinweis - en: Note - content: - de: "Bitte füllen Sie alle Pflichtfelder aus." - en: "Please fill in all required fields." - # oder dynamisch: - content: - eval: "`Zuletzt bearbeitet: ${$this.updateTime}`" - # oder aus externer Datei: - content: - path: - eval: "`/guides/${$this.type}.md`" -``` - -> ❌ **Nova-Status:** Nicht implementiert. - -### i18n - -Collection-spezifische i18n-Konfiguration. Sie erweitert oder überschreibt die projektweite `api.meta.i18n`-Konfiguration. Mit `false` kann i18n für eine Collection explizit deaktiviert werden. - -```yaml -meta: - i18n: - defaultLanguage: de - languages: - - code: de - label: - de: Deutsch - en: German - - code: en - label: - de: Englisch - en: English - entry: - languageField: lang - groupField: translationGroup - clearFields: [slug] -``` - -`entry.copyFields` ist alternativ zu `clearFields` möglich, wenn nur bestimmte Felder in neue Übersetzungen kopiert werden sollen. - -### sidebar - -Ordnet benannte Sidebar-Gruppen im Entry-Editor explizit und überschreibt bei Bedarf die Kartentitel. - -```yaml -meta: - sidebar: - - group: SEO - label: - de: Suchmaschinen-Optimierung - en: SEO - - group: Display -``` - -Das korrespondiert mit `meta.position: "sidebar:"` auf Feldebene. Ohne `label` rendert Nova die Sidebar-Karte ohne Titel. - -### sidebarGroups (deprecated) - -Legacy-Variante für Sidebar-Gruppen. Nova unterstützt sie noch, bevorzugt aber `sidebar` als geordnete Array-Form. - -```yaml -meta: - sidebarGroups: - SEO: - group: SEO - label: SEO -``` - -### sections - -Definiert benannte Sections im Hauptbereich des Entry-Editors für das neue Feld-Format `position: "section:"`. Sie ist das Section-Pendant zu [`sidebar`](#sidebar). - -```yaml -meta: - sections: - - group: seo - label: - de: Suchmaschinen-Optimierung - en: SEO - - group: advanced - label: Erweitert - collapsible: true # Überschrift wird zum Toggle - collapsed: false # startet aufgeklappt (Default) - - group: quiet - # kein label -> Section ohne Überschrift -``` - -| Feld | Typ | Default | Bedeutung | -|------|-----|---------|-----------| -| `group` | String | — | Schlüssel; muss zur Key-portion von `position: "section:"` passen | -| `label` | String + I18nText | Key-String | Überschrift. Ohne `label` keine Überschrift | -| `collapsible` | Boolean | `false` | Überschrift wird einklappbar | -| `collapsed` | Boolean | `false` | Startet eingeklappt (wirkt nur mit `collapsible`) | - -**Die Reihenfolge im `sections`-Array ist irrelevant** — sie steuert das Rendering nicht. Eine Section erscheint genau dort, wo ihr **erstes** Feld in der Feldreihenfolge steht; danach sammelt sie alle Felder mit demselben Group-Key ein, egal wie weit auseinander sie deklariert sind. Ein nicht deklarierter Key fällt auf den Key-String als Überschrift zurück. - -Felder ohne `position: "section:*"` werden **nicht** in eine Section hineingezogen — sie bleiben in ihrem eigenen Block, auch wenn sie zwischen zwei Feldern derselben Section stehen. - -### pagebuilder - -Collection-level Defaults für alle `widget: pagebuilder`-Felder der Collection. - -```yaml -meta: - pagebuilder: - blockTypeField: type - defaultViewport: 1280 - blockRegistry: - file: /assets/pagebuilder/blocks.mjs - screenshot: - - field: contentBlocks - fileField: previewImage - screenWidth: 1440 - imageWidth: 1200 -``` - -Feld-spezifische `meta.pagebuilder.*`-Werte überschreiben diese Collection-Defaults. - -`blockRegistry.file` ist der richtige Ort fuer projektspezifische Aufloesung. Der Wert kann per `{ eval: "..." }` mit `$projectBase` gebildet werden, damit immer das Registry-Modul des aktuellen Projekts geladen wird. - -Die CSS-Dateien innerhalb der Block-Registry sollten dagegen nicht erneut ueber `$projectBase` zusammengesetzt werden. Die Registry sollte fertige CSS-URLs exportieren, idealerweise relativ zum Registry-Modul selbst, z. B. `new URL("./hero.css", import.meta.url).toString()`. So bleiben Registry und CSS im selben Projekt verankert, und der Preview-Renderer kann die geladenen Stylesheets fuer Breakpoint-Umschreibung verarbeiten. - -Wichtig fuer die Toolchain: Das Build-Setup muss diese CSS-Dateien als echte statische Assets mit ausliefern. Bei esbuild darf die Referenz nicht wegoptimiert, in reines JS verschmolzen oder auf einen Pfad umgebogen werden, der neben dem Registry-Modul nicht mehr existiert. Die erzeugte Registry-Datei muss zur Laufzeit weiterhin eine aufloesbare CSS-URL liefern, damit Nova die Datei per `fetch()` laden und die Breakpoints umschreiben kann. - -`pagebuilder.screenshot` ist in Nova umgesetzt und erzeugt beim Speichern Vorschau-Bilder für konfigurierte Pagebuilder-Felder in ein File-Feld derselben Collection. - -### rowIdentTpl (deprecated) - -```yaml -meta: - # Deprecated — verwende stattdessen: - # preview: title - # preview: { eval: "..." } - rowIdentTpl: "{{ title }}" # Twig-Syntax - rowIdentTpl: - twig: "{{ firstName }} {{ lastName }}" - projection: teaser - rowIdentTpl: - eval: "`${$this.title} (${$this.type})`" -``` - -> 🗑️ **Deprecated:** Verwende `preview` stattdessen. Nova ignoriert `rowIdentTpl`. - -### tablist (deprecated) - -```yaml -meta: - # Deprecated — verwende stattdessen section auf Feld-Ebene - tablist: - activeTab: general - tabs: - - name: general - label: - de: Allgemein - en: General - muiIcon: cog - subFields: - - source: title - - source: slug - - name: seo - label: SEO - subFields: - - source: seo.title - - source: seo.description -``` - -> 🗑️ **Deprecated:** Nova verwendet `section` auf Feld-Ebene für Gruppierung. Siehe [section](#section). - -### quickEdit (deprecated) - -```yaml -meta: - # Deprecated — Nova hat Bulk-Edit - quickEdit: - enabled: true - fields: [title, status, sort] -``` - -> 🗑️ **Deprecated:** Nova verwendet Bulk-Edit, das mächtiger ist. - -### noDB (deprecated) - -```yaml -meta: - # Extrem seltenes Feature, wird nicht in Nova implementiert - noDB: - enabled: true - xmlPreview: true -``` - -> 🗑️ **Deprecated:** Wird nicht in Nova implementiert. - ---- - -## 4. Feld-Schema - -Jedes Feld hat einen Namen, Typ und optionale Metadaten. - -**Schema:** `tibi-types/schemas/api-config/field.json` -**Types:** `types/admin.d.ts → AdminCollectionField` - -```yaml -fields: - - name: title # Alphanumerisch + Unterstrich (required) - type: string # Datentyp (required) - index: [single, text] # Optional: DB-Index - validator: # Optional: Validierung - required: true - meta: # Optional: UI-Metadaten - label: - de: Titel - en: Title - - - name: content - type: object - readonly: - eval: "$method == \"put\"" - hidden: internalToggle - subFields: # Für object/object[] - - name: heading - type: string - - name: body - type: string - meta: - widget: richtext -``` - -### Name - -- Muss dem Pattern `^[0-9a-zA-Z_]+$` entsprechen -- Wird als Dokumentschlüssel in MongoDB verwendet -- Reservierte Präfixe: `_` (System-Felder wie `id`, `insertTime`, `updateTime`) - -### SubFields - -Für `object` und `object[]` Typen. Rekursiv verschachtelbar. - -```yaml -- name: sections - type: object[] - subFields: - - name: heading - type: string - - name: items - type: object[] - subFields: - - name: label - type: string - - name: url - type: string -``` - -`subFields` kann auch ein String sein, der auf eine gemeinsame Feld-Definition verweist: - -### readonly - -Top-level Feldkonfiguration für serverseitiges Readonly. Unterstützt `true`, `false`, String-Referenzen und `{ eval }`. - -```yaml -- name: slug - type: string - readonly: true - -- name: externalId - type: string - readonly: - eval: "$method == \"put\"" -``` - -Wichtig: - -- `readonly` ist **kein** `meta`-Feld, sondern liegt direkt auf dem Feld -- bei `true` auf Create und ohne Wert blendet Nova das Feld aus -- `readonly: false` kann vererbtes Config-Readonly eines Elternfelds aufbrechen - -### hidden - -Top-level Feldkonfiguration für serverseitiges Verbergen. Unterstützt `true`, String-Referenzen und `{ eval }`. - -```yaml -- name: internalNotes - type: string - hidden: true - -- name: billingComment - type: string - hidden: - eval: "$parent.type !== 'invoice'" -``` - -Admins sehen solche Felder in Nova standardmäßig mit Hidden-Hinweis und können sie temporär entsperren. - -```yaml -- name: content - type: object - subFields: contentFields # Referenz auf benannte Feld-Gruppe -``` - ---- - -## 5. Feld-Typen (Server) - -Der tibi-server unterstützt 10 Datentypen: - -| Typ | MongoDB-Typ | Beschreibung | Beispiel | -|-----|-------------|-------------|---------| -| `string` | String | Text | `"Hallo Welt"` | -| `string[]` | Array\ | Text-Array | `["tag1", "tag2"]` | -| `number` | Double/Int | Zahl | `42`, `3.14` | -| `number[]` | Array\ | Zahlen-Array | `[1, 2, 3]` | -| `boolean` | Boolean | Wahrheitswert | `true` | -| `object` | Object | Verschachteltes Objekt | `{ "key": "value" }` | -| `object[]` | Array\ | Objekt-Array | `[{ "a": 1 }, { "b": 2 }]` | -| `file` | Object | Datei-Metadaten | `{ "src": "...", "type": "image/jpeg", ... }` | -| `date` | Date | Datum/Zeit | `"2026-02-18T10:30:00Z"` | -| `any` | Mixed | Beliebiger Wert | JSON-Wert | - -### File-Objekt Struktur - -Dateien werden als Objekt gespeichert: - -```json -{ - "src": "/uploads/image.jpg", - "type": "image/jpeg", - "path": "image.jpg", - "size": 102400 -} -``` - ---- - -## 6. Widget-Typen (Admin UI) - -Widgets bestimmen, wie ein Feld im Admin-UI dargestellt wird. Wird kein Widget angegeben, wird der Standard-Widget basierend auf dem Feld-Typ verwendet. - -**Types:** `types/admin.d.ts → AdminWidgetType` - -### Widget-Mapping - -| Widget(s) | Komponente | Server-Typ | Beschreibung | -|-----------|-----------|-----------|-------------| -| `text`, `string`, `input` | TextWidget | `string` | Einzeiliges Textfeld | -| `imageURL` | TextWidget | `string` | Text mit Bild-Vorschau | -| `number`, `int`, `integer`, `float`, `double` | NumberWidget | `number` | Zahlenfeld | -| `bool`, `boolean`, `check`, `switch`, `checkbox` | BooleanWidget | `boolean` | Checkbox/Toggle | -| `select` | SelectWidget | `string \| number` | Dropdown-Auswahl | -| `selectArray` | SelectWidget | `string[] \| number[]` | Multi-Select Dropdown | -| `chipArray` | ChipArrayWidget | `string[] \| number[]` | Chip-Eingabe (Tags) | -| `checkboxArray` | ChipArrayWidget | `string[] \| number[]` | Checkbox-Gruppe | -| `date` | DateWidget | `date` | Datumsauswahl | -| `datetime` | DateWidget | `date` | Datum + Uhrzeit | -| `file` | FileWidget | `file` | Datei-Upload | -| `image` | FileWidget | `file` | Bild-Upload mit Vorschau, optionaler Editor & Fokuspunkt | -| `richtext`, `html` | RichtextWidget | `string` | WYSIWYG-Editor | -| `json`, `jsonField` | JsonWidget | `any` | JSON-Editor | -| `object` | ObjectField | `object` | Verschachteltes Objekt | -| `containerLessObject` | ObjectField | `object` | Objekt ohne Container, Label im Formular standardmäßig ausgeblendet | -| `objectArray` | ObjectArrayField | `object[]` | Array von Objekten | -| `containerLessObjectArray` | ObjectArrayField | `object[]` | Array ohne Container | -| `foreignKey` | ForeignKeyWidget | `string` | FK-Auswahl (einzeln) | -| `foreignKeyChipArray` | ForeignKeyChipArrayWidget | `string[]` | FK-Auswahl (mehrere) | - -Hinweis zu Labeln im Formular: - -- `meta.hideLabel: true` blendet die Label-Zeile im Entry-Editor aus -- `containerLessObject` verhält sich so, als wäre `meta.hideLabel: true` gesetzt -- In Collection-Views wie Tabelle, Cards oder Media-Previews bleibt die normale Anzeige unverändert - -### Nicht in Nova implementierte Widgets - -| Widget | Legacy-Status | Nova-Empfehlung | -|--------|-------------|-----------------| -| `contentbuilder` | ✅ Legacy | Kein Ersatz — Speziallösung | -| `grid` | ✅ Legacy | Kein Ersatz | -| `linkedEntries` | ✅ Legacy | Kein Ersatz | -| `tabs` | 🗑️ Deprecated | → `section` auf Feldern | -| `previewBasedObjectArray` | 🗑️ Deprecated | → `drillDown: true` | - -### Beispiele - -```yaml -# Einfaches Textfeld -- name: title - type: string - meta: - widget: text # Optional, Standard für string - -# Richtext-Editor -- name: content - type: string - meta: - widget: richtext - -# Datum mit Uhrzeit -- name: publishDate - type: date - meta: - widget: datetime - -# Select mit Choices -- name: status - type: string - meta: - widget: select - choices: - - id: draft - name: Entwurf - - id: published - name: Veröffentlicht - -# Chip-Array Tags -- name: tags - type: string[] - meta: - widget: chipArray - -# Foreign Key -- name: author - type: string - meta: - widget: foreignKey - foreign: - collection: authors - render: name - -# Bild mit Editor und Auto-Downscale -# (imageEditor ist standardmäßig aktiv für alle image-Widgets, kann aber konfiguriert werden) -- name: cover - type: file - meta: - widget: image - label: Titelbild - # imageEditor: true # Default — muss nicht explizit gesetzt werden - downscale: # Auto-Downscale vor Upload - maxWidth: 2000 - maxHeight: 2000 - quality: 0.85 - -# Bild mit selektiven Editor-Tools -- name: thumbnail - type: file - meta: - widget: image - imageEditor: - crop: true - focalPoint: true - rotate: false - flip: false - adjust: false - -# Bild ohne Editor -- name: photo_no_editor - type: file - meta: - widget: image - imageEditor: false # Editor explizit deaktivieren - -# Bild nur mit Downscale (Editor bleibt aktiv) -- name: photo - type: file - meta: - widget: image - downscale: - maxWidth: 1600 - quality: 0.8 -``` - ---- - -## 7. Feld-Meta - -Steuert Darstellung und Verhalten eines Feldes im Admin-UI. - -**Schema:** `tibi-types/schemas/api-config/fieldMeta.json` -**Types:** `types/admin.d.ts → AdminCollectionFieldMeta` - -### Übersicht aller Feld-Meta-Optionen - -| Option | Typ | Nova | Beschreibung | -|--------|-----|------|-------------| -| [`label`](#label-1) | `string \| I18nText` | ✅ | Anzeigename | -| [`widget`](#widget) | `AdminWidgetType` | ✅ | Widget-Typ | -| [`i18n`](#i18n-1) | `false` | ✅ | i18n-Autoerkennung deaktivieren | -| [`helperText`](#helpertext) | `string \| I18nText` | ✅ | Hilfetext unter dem Feld | -| [`hideLabel`](#hidelabel) | `boolean` | ✅ | Nur Label-Zeile im Formular ausblenden | -| [`hide`](#hide) | `boolean` | ✅ | Feld ausblenden | -| [`protected`](#protected) | `boolean \| string \| { eval }` | ✅ | Soft-Readonly mit Entsperren | -| [`position`](#position) | `"main" \| "sidebar" \| "sidebar:" \| "section" \| "section:"` | ✅ | Position im Editor | -| [`section`](#section-legacy) | `string` | ✅ | Legacy-Gruppierung unter Überschrift | -| [`containerProps`](#containerprops) | `object` | ✅ | Layout (Grid-Spalten, CSS-Klasse) | -| [`defaultValue`](#defaultvalue) | `any \| { eval }` | ✅ | Standardwert für neue Einträge | -| [`dependsOn`](#dependson) | `string \| { eval }` | ✅ | Bedingte Sichtbarkeit | -| [`inputProps`](#inputprops) | `Record` | ✅ | Widget-spezifische Props | -| [`choices`](#choices) | `Choice[] \| Endpoint` | ✅ | Auswahloptionen | -| [`foreign`](#foreign) | `ForeignConfig` | ✅ | Foreign-Key-Konfiguration | -| [`downscale`](#downscale) | `object` | ✅ | Auto-Downscale vor Bild-Upload | -| [`imageEditor`](#imageeditor) | `boolean \| object` | ✅ | Bildeditor-Tools für Image-Widgets | -| [`drillDown`](#drilldown) | `boolean` | ✅ | Drill-Down-Navigation | -| [`pageSize`](#pagesize) | `number` | ✅ | Pagination für object[] | -| [`preview`](#preview-feld) | `string \| { eval } \| PreviewSlots` | ✅ | Vorschautext für Array-Items | -| [`addElementLabel`](#addelementlabel) | `string \| I18nText` | ⚠️ | "Hinzufügen"-Button-Text | -| [`filter`](#filter-feld) | `boolean \| { type, facets }` | ✅ | Filter-Typ in Listenansicht | -| [`pagebuilder`](#pagebuilder-1) | `object` | ✅ | Pagebuilder-Feldkonfiguration | -| [`css`](#css) | `AdminFieldCss` | ✅ | CSS-Overrides, Eval-Variante deprecated | -| [`folding`](#folding) | `object` | 🗑️ | → `drillDown` | -| [`metaElements`](#metaelements) | `various` | 🗑️ | → `section` | -| [`pathStep`](#pathstep) | `object` | 🗑️ | → automatische Breadcrumbs | -| [`direction`](#direction) | `string` | 🗑️ | → immer vertikal + drillDown | -| [`valueMap`](#valuemap) | `Record` | 🗑️ | → `choices[].chipStyle` | -| [`openapi`](#openapi) | `Record` | — | Nur API-Docs, kein UI | - ---- - -### label {#label-1} - -```yaml -meta: - label: Titel - # oder mehrsprachig: - label: - de: Titel - en: Title -``` - -### widget - -Überschreibt den automatisch erkannten Widget-Typ. - -```yaml -meta: - widget: richtext # Statt Standard "text" für string -``` - -### helperText - -Hilfetext unterhalb des Eingabefelds. - -```yaml -meta: - helperText: - de: "Max. 160 Zeichen für SEO" - en: "Max. 160 characters for SEO" -``` - -### hideLabel - -Blendet nur die Label-Zeile im Entry-Editor aus. Das Feld selbst bleibt sichtbar, und Collection-Ansichten wie Tabelle, Karten oder sonstige Previews verwenden das Label weiterhin normal. - -```yaml -meta: - hideLabel: true -``` - -Standardverhalten: - -- `containerLessObject` setzt `hideLabel` implizit auf `true` -- Alle anderen Widgets/Feldtypen verwenden standardmäßig `false` -- Mit `hideLabel: false` kann das Default-Verhalten von `containerLessObject` explizit überschrieben werden - -### i18n - -Mit `false` kann die automatische i18n-Erkennung für dieses Feld deaktiviert werden, selbst wenn Sub-Fields Sprachcodes entsprechen. - -```yaml -meta: - i18n: false -``` - -### hide - -Blendet das Feld komplett aus dem Editor aus. - -```yaml -meta: - hide: true -``` - -### protected - -Clientseitiges Soft-Readonly. Das Feld ist zunächst gesperrt, kann aber in Nova per Badge entsperrt werden. - -```yaml -meta: - protected: true - - # Oder dynamisch: - protected: - eval: "$method == 'put'" -``` - -### position - -Position im Entry-Editor: Hauptbereich, Sidebar oder Section. - -```yaml -meta: - position: sidebar # oder "main" (default) - position: "sidebar:SEO" # benannte Sidebar-Karte - position: "section:seo" # benannte Section (Titel via meta.sections) - position: section # Section-Gruppe ohne Key -``` - -### section (Legacy) - -Legacy-Format: gruppiert Felder unter einer Überschrift. Der Wert **ist** die Überschrift, die Gruppierung ist rein sequenziell — nur direkt benachbarte Felder mit gleichem Namen bilden eine Gruppe, und ein Feld ohne `section` läuft in der aktuellen Gruppe weiter. Ersetzt die Legacy `tablist`-Konfiguration. - -```yaml -# Alle Felder mit gleicher section werden zusammengefasst: -- name: seoTitle - type: string - meta: - section: SEO - label: SEO-Titel - -- name: seoDescription - type: string - meta: - section: SEO - label: SEO-Beschreibung - widget: text - inputProps: - multiline: true - rows: 3 -``` - -**Empfehlung:** Für neue Collections das [neue Format](#section-neues-format) verwenden. Es erlaubt i18n-Überschriften über [`meta.sections`](#sections), `collapsible`/`collapsed` und fasst auch nicht benachbarte Felder zusammen. - -#### Section (neues Format) - -```yaml -meta: - sections: - - group: seo - label: Suchmaschinen-Optimierung - - group: advanced - label: Erweitert - collapsible: true - collapsed: true - -fields: - # Legt die Section an der Position dieses Feldes an … - - name: seoTitle - meta: - position: "section:seo" - - # … und zieht dieses hier ebenfalls nach, auch wenn es weiter unten steht. - - name: seoDescription - meta: - position: "section:seo" -``` - -Die beiden Formate sind **vollständig getrennt**: `section: SEO` (Legacy) und `position: "section:SEO"` (neu) erzeugen auch dann zwei getrennte Gruppen, wenn die Überschriften identisch aussehen. - -### containerProps - -Layout im 12-Spalten CSS-Grid. - -```yaml -meta: - containerProps: - class: "my-custom-class" # Zusätzliche CSS-Klasse - layout: - size: "col-6" # Halbe Breite (2 Felder nebeneinander) - # size: "col-4" # Ein Drittel (3 Felder pro Zeile) - # size: "col-3" # Ein Viertel (4 Felder pro Zeile) - # size: "col-12" # Volle Breite (Standard) - - # Responsive (mobile-first): - # Bei small+default+large: small=Basis, default=md:, large=lg: - # Bei default+large: default=Basis, large=lg: - size: - default: "col-12" - small: "col-6" - large: "col-4" - - breakBefore: true # Neue Zeile davor erzwingen - breakAfter: true # Neue Zeile danach erzwingen -``` - -### defaultValue - -Standardwert für neue Einträge. - -```yaml -meta: - # Statischer Wert: - defaultValue: draft - - # Dynamischer Wert: - defaultValue: - eval: "new Date().toISOString()" - - # Für object[]: - defaultValue: - eval: "[{ heading: '', body: '' }]" -``` - -### dependsOn - -Bedingte Sichtbarkeit. Feld wird nur angezeigt, wenn die Bedingung wahr ist. - -```yaml -meta: - # Einfach: Feld ist sichtbar wenn "showBanner" truthy ist - dependsOn: showBanner - - # Dot-Notation für verschachtelte Felder: - dependsOn: settings.enableSeo - - # Eval für komplexe Logik: - # Variablen: $ = Root-Entry, $this = aktuelles Feld, $parent, $stack - dependsOn: - eval: "$.type === 'special' && $parent.active" -``` - -### inputProps - -Widget-spezifische Properties. - -```yaml -meta: - # Mehrzeiliges Textfeld: - inputProps: - multiline: true - rows: 5 - - # Zahlenfeld mit Einschränkungen: - inputProps: - min: 0 - max: 100 - step: 0.5 - - # Placeholder: - inputProps: - placeholder: "https://..." -``` - -### choices - -Siehe [Choices-Konfiguration](#8-choices-konfiguration). - -### foreign - -Siehe [Foreign-Key-Konfiguration](#9-foreign-key-konfiguration). - -### downscale - -Clientseitiges automatisches Herunterskalieren von Bildern vor dem Upload. - -```yaml -meta: - widget: image - downscale: - maxWidth: 2000 - maxHeight: 2000 - quality: 0.85 -``` - -### imageEditor - -Aktiviert den clientseitigen Bildeditor für Bildfelder. - -```yaml -meta: - widget: image - imageEditor: true - - # Oder selektiv: - imageEditor: - crop: true - rotate: true - flip: true - focalPoint: true - adjust: true - palette: true -``` - -### drillDown - -Steuert, ob verschachtelte Objekte/Arrays inline oder via Drill-Down-Navigation dargestellt werden. - -```yaml -meta: - # Force Drill-Down (auch auf Root-Level): - drillDown: true - - # Force Inline (auch in tiefer Verschachtelung): - drillDown: false - - # Auto (Standard): Root → Inline, tiefer → Drill-Down - # (drillDown weglassen) -``` - -### preview (Feld) {#preview-feld} - -Vorschautext für Array-Items und verschachtelte Objekte. - -```yaml -meta: - # Wert eines Sub-Felds als Vorschau: - preview: headline - - # Dynamisch: - preview: - eval: "`${$this.type}: ${$this.title}`" -``` - -Falls nicht gesetzt, wird das erste String-Sub-Feld als Vorschau verwendet. - -Zusätzlich unterstützt Nova auch strukturierte Preview-Slots analog zu `collection.meta.preview`, z. B. `label`, `secondary`, `badge`, `image` oder explizite `table`-Definitionen für Object-/Readonly-Previews. - -### addElementLabel - -Text für den "Element hinzufügen"-Button bei Arrays. - -```yaml -meta: - addElementLabel: - de: Abschnitt hinzufügen - en: Add section -``` - -> ⚠️ **Nova-Status:** Noch hardcoded. - -### pageSize - -Paginator für große `object[]`-Felder. Standard ist `20` sichtbare Elemente. - -```yaml -meta: - pageSize: 10 - - # Alle Elemente sofort anzeigen: - pageSize: 0 -``` - -### filter (Feld) {#filter-feld} - -Filter-Typ für dieses Feld in der Listenansicht-Filterleiste. - -```yaml -meta: - filter: - type: select # default | select | bool | boolean | input | number | foreignKey - facets: true # Facettierte Werte über _facet laden - - # Oder einfacher Opt-in/Opt-out: - filter: true -``` - -Top-Level-Felder erscheinen in Nova standardmäßig in der Filterleiste, sofern ihr Typ sinnvoll filterbar ist. Für Sub-Felder von `object`/`object[]` dient `meta.filter` als explizites Opt-in. - -### pagebuilder - -Feld-spezifische Pagebuilder-Konfiguration. - -```yaml -meta: - widget: pagebuilder - pagebuilder: - defaultViewport: tablet - blockTypeField: type - blockRegistry: - file: /assets/pagebuilder/landing-blocks.mjs -``` - -In den aktuellen Admin-Typen und im Nova-Renderpfad sind hier `defaultViewport`, `blockTypeField` und `blockRegistry.file` relevant. - -Fuer projektspezifische Setups sollte `blockRegistry.file` das aktuelle Projekt ueber `$projectBase` adressieren, waehrend die Block-Registry ihre eigenen CSS-Dateien relativ zu `import.meta.url` aufloest. Beispiel: - -```yaml -meta: - widget: pagebuilder - pagebuilder: - blockRegistry: - file: - eval: "`${$projectBase}assets/pagebuilder/registry.mjs`" -``` - -```ts -const heroCss = new URL("./hero.css", import.meta.url).toString() - -export default { - hero: { - css: [heroCss], - render(container, row, context) { - return { - update(nextRow, nextContext) {}, - destroy() {}, - } - }, - }, -} -``` - -So ist der Projektbezug nur an einer Stelle konfiguriert, und die CSS-Dateien bleiben robust relativ zum Registry-Modul referenzierbar. - -Auch hier gilt: Die Toolchain muss `new URL("./hero.css", import.meta.url)` als Asset-Referenz erhalten. Wenn ihr mit esbuild bundelt, muss das CSS als Datei im Output landen; ein Bundle, das die Styles nur inline in JS einbettet oder unbenutzte Asset-URLs entfernt, funktioniert fuer den Pagebuilder-Preview nicht. - -### css - -CSS-Overrides für das Feld. - -```yaml -meta: - css: "background-color: #f0f0f0" - - # Oder detailliert: - css: - input: - wrapper: "border: 2px solid blue" - element: "font-size: 1.2em" - view: - wrapper: "padding: 8px" -``` - -> ⚠️ **Nova-Status:** Teilweise, `{ eval }` Variante deprecated (Sicherheitsrisiko durch eval + @emotion/css). - -### folding (deprecated) - -```yaml -meta: - # Deprecated — verwende drillDown: true/false stattdessen - folding: - force: true - unfolded: false - previewFolded: title - previewUnfolded: - eval: "`${$this.title} — ${$this.items?.length || 0} Items`" -``` - -### metaElements (deprecated) - -```yaml -meta: - # Deprecated — verwende section auf Sub-Feldern stattdessen - metaElements: - tablist: - tabs: - - name: general - label: Allgemein - subFields: - - source: title - - source: slug - - name: seo - label: SEO - subFields: - - source: seoTitle - - source: seoDescription -``` - -### pathStep (deprecated) - -```yaml -meta: - # Deprecated — Nova generiert Breadcrumbs automatisch aus label + preview - pathStep: - title: - de: Abschnitte - en: Sections - icon: view-list -``` - -### direction (deprecated) - -```yaml -meta: - # Deprecated — Nova verwendet immer vertikal + Drill-Down - direction: horizontal # oder: vertical (default) -``` - -### valueMap (deprecated) - -```yaml -meta: - # Deprecated — verwende choices[].chipStyle stattdessen - valueMap: - active: - muiIcon: check - style: - color: green - inactive: - muiIcon: close - style: - color: red -``` - -### openapi - -```yaml -meta: - # Nur für API-Dokumentation, kein Einfluss auf Admin-UI - openapi: - description: "Eindeutiger Slug für die URL" - example: "/ueber-uns" -``` - ---- - -## 8. Choices-Konfiguration - -Auswahloptionen für `select`, `selectArray`, `chipArray`, `checkboxArray` Widgets. - -### Statisches Array - -```yaml -meta: - widget: select - choices: - - id: draft - name: - de: Entwurf - en: Draft - - id: published - name: - de: Veröffentlicht - en: Published - badgeStyle: - background: "#4caf50" - color: white - boxShadow: "inset 0 0 0 1px rgba(255,255,255,.18)" - - id: archived - name: Archiviert - chipStyle: - bg: "#9e9e9e" - text: "#ffffff" -``` - -| Feld | Typ | Beschreibung | -|------|-----|-------------| -| `id` | `string \| number \| null` | Wert der gespeichert wird | -| `name` | `string \| I18nText` | Anzeigename | -| `badgeVariant` / `chipVariant` | `"success" \| "error" \| "warning" \| "info" \| "neutral"` | Semantische Badge-Farbe ohne eigenes CSS | -| `badgeStyle` / `chipStyle` | `Record` | Flache Inline-CSS-Properties für Badge-/Chip-Darstellung | - -`badgeStyle` und `chipStyle` überschreiben `badgeVariant` beziehungsweise `chipVariant`. -Erlaubt sind flache Inline-Style-Properties mit skalaren Werten, also zum Beispiel `background`, `color`, `borderColor`, `boxShadow`, `textTransform` oder CSS-Custom-Properties wie `--badge-border`. -CamelCase und kebab-case funktionieren beide. -Die Kurzschreibweisen `bg` und `text` bleiben als Aliasse für `background` und `color` erhalten. -Nicht unterstützt sind verschachtelte Objekte, Pseudo-Selektoren wie `:hover` oder Media Queries. - -### Dynamischer Endpoint - -Lädt Choices von einer API. - -```yaml -meta: - widget: select - choices: - endpoint: /api/my_project/categories - mapping: - id: _id # Welches Feld als Wert verwenden - name: title # Welches Feld als Anzeigename - params: - _sort: name - _limit: 100 -``` - ---- - -## 9. Foreign-Key-Konfiguration - -Verknüpft ein Feld mit Einträgen einer anderen Collection. - -**Types:** `types/admin.d.ts → AdminFieldForeign` - -```yaml -- name: author - type: string - meta: - widget: foreignKey - foreign: - collection: authors # Ziel-Collection (required) - id: _id # ID-Feld (default: "_id") - filter: { active: true } # Optionaler statischer Filter - sort: -insertTime # Sortierung ("-" für DESC) - projection: teaser # Benannte Projektion - render: name # Anzeige: Feldname - subNavigation: 0 # Index in subNavigation der Ziel-Collection - -# Multi-Select Foreign Key: -- name: categories - type: string[] - meta: - widget: foreignKeyChipArray - foreign: - collection: categories - render: - eval: "`${$this.name} (${$this.count} Einträge)`" - raw: false # true = HTML erlauben -``` - -### Foreign Render-Optionen - -```yaml -foreign: - # Einfach: Feldname - render: title - - # Eval: JS-Ausdruck mit $this = Foreign-Entry - render: - eval: "`${$this.firstName} ${$this.lastName}`" - - # Raw HTML (eval): - render: - eval: "`${$this.title}
${$this.category}`" - raw: true - - # Collection-Views der Ziel-Collection verwenden: - render: - defaultCollectionViews: true - - # Strukturierte Preview-Slots: - render: - label: name - secondary: email - badge: status - image: avatar -``` - -### Foreign Filter - -```yaml -foreign: - # Statisch: - filter: - active: true - - # Dynamisch: gleicher Eval-Kontext wie bei dependsOn - filter: - eval: | - $.id - ? { userId: $.id } - : { userId: "__no_match__" } -``` - -Der Filter wirkt sowohl in der Inline-Suche als auch im Browse-Drawer. - -### Create Defaults (Inline-Create Pre-Fill) - -Beim Inline-Erstellen im FK-Drawer werden automatisch einfache Werte aus dem aufgelösten `filter` und der aktiven SubNavigation als Vorbelegung übernommen. Zusätzlich kann `createDefaults` explizite Werte definieren. - -**Merge-Priorität:** `auto-extracted(filter)` → `auto-extracted(subNavFilter)` → `createDefaults` - -```yaml -foreign: - collection: people - filter: - position: CTO # → position wird automatisch vorbelegt - createDefaults: - department: Engineering # → zusätzlich department vorbelegen - - # Dynamisch (eval): - createDefaults: - eval: | - $parent.defaultDepartment - ? { department: $parent.defaultDepartment } - : {} -``` - -> **Auto-Extraction:** Nur primitive Werte (string, number, boolean) werden extrahiert. Keys mit `$`-Prefix und Objekt/Array-Werte werden übersprungen. - -### foreign.id — Kompatibilitätshinweis - -Der Server akzeptiert sowohl `_id` als auch `id` in Filter- und Select-Queries. -Daher funktionieren alte und neue Configs gleichermaßen. - -| Admin | Default `foreign.id` | Bemerkung | -|-------|---------------------|----------| -| Legacy (tibi-admin) | `"id"` | Server akzeptiert beides | -| Nova (tibi-admin-nova) | `"_id"` | Server akzeptiert beides | - -> **Empfehlung:** Explizit `id: "_id"` setzen für Klarheit. - ---- - -## 10. Validator - -Validierungsregeln für Felder. - -**Schema:** `tibi-types/schemas/api-config/field.json → validator` - -```yaml -validator: - required: true # Pflichtfeld (Server + Client) - allowZero: true # "" oder 0 erlauben bei required - minLength: 3 # Min. Zeichenlänge (nur Client) - maxLength: 160 # Max. Zeichenlänge (nur Client) - pattern: "^[a-z0-9-]+$" # Regex-Pattern (nur Client) - eval: | # Server-seitiges JS - if ($this.length < 3) return "Mindestens 3 Zeichen" - if (!/^[a-z]/.test($this)) return "Muss mit Kleinbuchstabe beginnen" -``` - -| Option | Server | Client | Beschreibung | -|--------|--------|--------|-------------| -| `required` | ✅ | ✅ | Feld muss einen Wert haben | -| `allowZero` | ✅ | ✅ | Leerer String / 0 erlaubt bei required | -| `minLength` | ❌ | ✅ | Minimale String-Länge | -| `maxLength` | ❌ | ✅ | Maximale String-Länge | -| `pattern` | ❌ | ✅ | Regex-Validierung | -| `eval` | ✅ | ⚠️ | JS-Ausdruck (Client zeigt API-Fehler) | - -### Eval-Variablen - -| Variable | Beschreibung | -|----------|-------------| -| `$this` | Aktueller Feldwert | -| `$parent` | Eltern-Objekt | -| `$stack` | Array aller Eltern bis Root | -| `$auth` | Authentifiziertes User-Objekt | -| `context` | Server-Kontext (db, config, etc.) | - ---- - -## 11. Search (Collection-Level) - -Benannte Suchkonfigurationen für die Nova-Suche. Wenn mehrere Einträge definiert sind, kann Nova zwischen Suchmodi umschalten. - -```yaml -search: - - name: Standard - mode: text - fields: [title, excerpt, body] - - - name: Filter - mode: filter - filter: - status: published - - - name: Regex - mode: regex - fields: [title, slug] - regex: - scoring: count - weights: - title: 2 - slug: 1 -``` - -Unterstützte Modi laut Schema: - -- `text` -- `regex` -- `eval` -- `filter` -- `ngram` -- `vector` -- `combined` - -Je nach Modus können zusätzliche Blöcke wie `eval`, `filter`, `ngram`, `regex`, `score` oder mode-spezifische `meta`-Optionen verwendet werden. - -## 12. Views (List-Ansichten) - -Definieren, wie Einträge in der Listenansicht dargestellt werden. - -**Schema:** `tibi-types/schemas/api-config/collectionNavigation.json → views` - -> ⚠️ **Nova-Status:** Nova hat eine eingebaute responsive Tabelle + Cards-Toggle. Legacy-View-Definitionen werden aktuell nicht ausgewertet. - -### Table View - -```yaml -meta: - views: - - type: table - mediaQuery: "(min-width: 768px)" # 🗑️ Nova ignoriert - selectionPriority: 1 # 🗑️ Nova ignoriert - columns: - - source: title - label: - de: Titel - en: Title - width: "40%" - - source: status - label: Status - width: "15%" - align: center - - source: insertTime - label: Erstellt - width: "20%" - eval: "new Date($this).toLocaleDateString('de')" -``` - -### Simple List View - -```yaml -meta: - views: - - type: simpleList - primaryText: title - secondaryText: - eval: "`${$this.category} — ${new Date($this.insertTime).toLocaleDateString('de')}`" - tertiaryText: status - image: cover.src -``` - -### Card List View - -```yaml -meta: - views: - - type: cardList - fields: - - source: cover - type: image - label: Bild - - source: title - label: Titel - - source: status - label: Status - widget: chip -``` - -### Dashboard-Varianten - -`dashboardTable` und `dashboardSimpleList` funktionieren gleich, werden aber im Dashboard-Kontext angezeigt. - -### File Drop Area - -Kann an jede View angehängt werden: - -```yaml -meta: - views: - - type: table - fileDropArea: - fields: - - source: file - prefilledFields: - - source: title - defaultValue: - eval: "$file.name.replace(/\\.[^.]+$/, '')" -``` - ---- - -## 13. Kompatibilitätsmatrix Legacy → Nova - -### Ersetzungen (Migration empfohlen) - -| Legacy | Nova-Ersatz | Migrations-Aufwand | -|--------|------------|-------------------| -| `tablist` auf Collection-Meta | `section` auf jedem Feld | Gering — `section: "TabName"` pro Feld | -| `tabs` Widget | `section` auf Sub-Feldern | Gering | -| `previewBasedObjectArray` Widget | `drillDown: true` | Minimal — Property umbenennen | -| `folding` | `drillDown: true/false` | Minimal | -| `quickEdit` | Nova Bulk-Edit (automatisch) | Keine Aktion nötig | -| `rowIdentTpl` | `preview` | Syntax ändern (twig → eval/string) | -| `direction: horizontal` | `chipArray` Widget | Je nach Kontext | -| `valueMap` | `choices[].chipStyle` | Styles in choices verschieben | -| `metaElements` | `section` auf Sub-Feldern | Gering | -| `pathStep` | Automatische Breadcrumbs | Entfernen | -| `css.*.eval` (Emotion) | `containerProps.class` + Tailwind | Logik umschreiben | - -### Nicht implementiert (keine Migration) - -| Feature | Beschreibung | Workaround | -|---------|-------------|-----------| -| `contentbuilder` | Visueller Block-Editor | Kein Ersatz in Nova | -| `grid` | Raster-Layout Widget | Kein Ersatz | -| `linkedEntries` | Bidirektionale Verknüpfung | Kein Ersatz | -| `multiupload` | Multi-Datei-Upload | Einzeln hochladen | -| `backup` | Entry-Versionierung | — | -| `multilingual` | Mehrsprachigkeit | — | -| `noDB` | Kein-DB-Modus | — | -| `informationGuide` | Info-Box im Editor | — | -| `navigation.elements` | Custom Navigation | — | -| Custom `views` | Konfigurierbare Listenansichten | Nova auto-responsive | -| `MANUALLY` Sort | Drag-to-reorder | ⚠️ TODO | -| `defaultCallback: { eval }` | Custom Klick-Aktion | ⚠️ TODO | - -### Was funktioniert aber ignoriert wird - -Alte Configs mit diesen Properties brechen **nicht** — sie werden einfach ignoriert: - -- `tablist` — Felder werden ohne Tabs dargestellt -- `views` — Nova nutzt eigene responsive Tabelle -- `quickEdit` — Nova hat Bulk-Edit -- `mediaQuery` / `selectionPriority` — Nova auto-responsive -- `folding` — Nova nutzt auto-Drill-Down-Logik -- `rowIdentTpl` — Fallback auf `preview` oder `id` -- `noDB`, `backup`, `multilingual` — Features fehlen schlicht - ---- - -## 14. Bekannte Inkompatibilitäten & TODOs - -### System-Felder: `id`, `insertTime`, `updateTime` - -Der tibi-server gibt Collection-Einträge mit folgenden System-Feldern zurück: - -```json -{ - "id": "507f1f77bcf86cd799439011", - "insertTime": "2026-02-18T10:30:00Z", - "updateTime": "2026-02-18T12:00:00Z", - "title": "Mein Eintrag" -} -``` - -| Feld | Beschreibung | Typ | -|------|-------------|-----| -| `id` | MongoDB ObjectID als String | `string` | -| `insertTime` | Erstellzeitpunkt (vom Server gesetzt) | `string` (ISO 8601) | -| `updateTime` | Letzter Änderungszeitpunkt (vom Server gesetzt) | `string` (ISO 8601) | - -**Filter-Queries:** Der Server akzeptiert sowohl `_id` als auch `id` als Filter-Key. -Beide verweisen auf die MongoDB `_id`. Das erlaubt Kompatibilität mit alten Configs. - -**Hinweis:** Mock-Daten und Nova-Frontend verwenden aktuell `id`/`insertTime`/`updateTime` — -identisch zum Server-Format. ✅ - -### 🟡 `defaultSort` Format-Hinweis - -Das tibi-types Schema definiert `defaultSort` als Objekt: - -```yaml -defaultSort: - field: publishDate - order: DESC -``` - -Manche Legacy-Configs nutzten eine Kurzschreibweise (`"-publishDate"`), die vom alten Admin intern konvertiert wurde. -Nova sollte idealerweise beide Formate tolerieren. Aktuell werden nur Objekt-Formate korrekt verarbeitet. - -### 🟡 `foreign.id` — Server akzeptiert beide Varianten - -Der Server akzeptiert sowohl `_id` als auch `id` in Filter-Queries. Daher funktionieren alte -Configs (Default `"id"`) und neue Configs (Default `"_id"`) gleichermaßen. - -### 🟡 Tatsächliche Restpunkte in Nova - -Noch offen oder bewusst nicht umgesetzt sind derzeit nur die Punkte, die im Renderpfad bzw. in den Typ-Kommentaren selbst als offen markiert sind: - -- `defaultSort.order: MANUALLY` -- `defaultCallback` außerhalb des Standardverhaltens `edit` -- `defaultImageFilter` -- `collection.meta.filter` -- Custom `meta.views` -- `addElementLabel` - -### 🟢 Korrekt implementierte Features - -Folgende Features sind vollständig kompatibel mit dem tibi-server Schema: - -- `label` (String + I18nText) -- `muiIcon` -- `singleton` -- `hide` -- `subNavigation` mit `filter` -- `subNavigation.defaultSort` und `subNavigation.setDefault` -- `group` -- `preview` auf Collection-Ebene inkl. Eval und Slot-Varianten -- `i18n`, `sidebar`, `sidebarGroups` (Legacy), `sections`, `pagebuilder` auf Collection-Meta -- Alle 10 Feld-Typen -- Alle implementierten Widget-Typen (22+) -- `validator` (inkl. min/max, pattern, Datei-/Array-Constraints; `eval` serverseitig) -- `choices` (statisch + Endpoint) -- `foreign` inkl. `filter`, `render`, `subNavigation`, `createDefaults` -- `dependsOn` (String + eval) -- `defaultValue` (statisch + eval) -- `section` (Legacy) und `position: section:` (mit `meta.sections`) -- `containerProps` (responsive size + breakBefore/breakAfter) -- `inputProps` -- `hideLabel`, `protected`, `downscale`, `imageEditor` -- `drillDown` -- `pageSize` -- `preview` auf Feld-Ebene inkl. Eval und Slot-Varianten -- `filter` auf Feld-Ebene -- `position` (main/sidebar/sidebar:/section/section:) -- Top-level `readonly` und `hidden` -- `hide` -- `helperText` - ---- - -## Anhang: I18nText Format - -Mehrsprachige Texte können als einfacher String oder als Objekt mit Sprachcodes angegeben werden: - -```yaml -# Einfach: -label: Titel - -# Mehrsprachig: -label: - de: Titel - en: Title - fr: Titre -``` - -## Anhang: Eval-Ausdrücke - -Viele Konfigurationsoptionen unterstützen dynamische JS-Ausdrücke via `eval`: - -```yaml -# Template Literal: -eval: "`${$this.firstName} ${$this.lastName}`" - -# Bedingung: -eval: "$this.type === 'premium' ? 'Premium' : 'Standard'" - -# Zugriff auf Root-Entry: -eval: "$.title + ' — ' + $.status" -``` - -### Verfügbare Variablen in Eval - -| Variable | Kontext | Beschreibung | -|----------|---------|-------------| -| `$this` | Überall | Aktueller Wert / Entry | -| `$` | Entry-Kontext | Root-Entry-Objekt | -| `$parent` | Verschachtelt | Eltern-Objekt | -| `$stack` | Verschachtelt | Array aller Eltern | -| `$file` | multiupload | Upload-Datei-Objekt | -| `$auth` | Server-Hooks | Authentifizierter User | -| `context` | Server-Hooks | Server-Kontext (db, config, etc.) | -| `parent` | defaultCallback | Parent-Window (Modal-Kontext) | - -## Anhang: YAML !include - -Collections können in separate Dateien ausgelagert werden: - -```yaml -# config.yml -collections: - - !include collections/pages.yml - - !include collections/blog.yml -``` - -```yaml -# collections/pages.yml -name: pages -permissions: - get: public -fields: - - !include fields/title.yml - - !include fields/slug.yml - - name: content - type: string - meta: - widget: richtext -``` - -Auch Felder und Feld-Listen können per `!include` eingebunden werden: - -```yaml -# fields/title.yml -name: title -type: string -validator: - required: true -meta: - label: - de: Titel - en: Title -``` diff --git a/src/content/docs/nova/collection-export.md b/src/content/docs/nova/collection-export.md deleted file mode 100644 index f50d55b..0000000 --- a/src/content/docs/nova/collection-export.md +++ /dev/null @@ -1,331 +0,0 @@ ---- -title: 'Export-Funktion für Listenansichten (CSV / XLSX / JSON)' ---- - -# Export-Funktion für Listenansichten (CSV / XLSX / JSON) - -> Status: **implementiert** (2026-08-17) — alle Phasen abgeschlossen, 10/10 Playwright-Tests grün, `yarn validate` grün. -> Ziel: Export der aktuell gefilterten Liste (alle Seiten, nicht nur die aktuelle Seite) als CSV, XLSX oder JSON, mit vorgeschalteter Feldauswahl. - ---- - -## 1. Ziel & Umfang - -- Export der **aktuell gefilterten Liste** (Suchquery, Feld-Filter, aktiver SubNavigation) über **alle Seiten** hinweg. -- **Drei Formate:** JSON (Default), CSV, XLSX. -- **Vorgeschaltete Feldauswahl** im Dialog — nicht nur die aktuell sichtbaren Spalten, sondern (optional) alle Felder. -- **Batch-Laden** mit Fortschrittsanzeige, da der Server `limit` begrenzen kann. -- **Count-Warnung** bei sehr großen Sammlungen. -- **Per-Feld konfigurierbare `object[]`-Behandlung** (nur im CSV/XLSX-Modus relevant). -- Wiederverwendung bestehender UI-Patterns und Utilities (DRY). - -**Nicht im Scope:** -- Relationaler „Pivot"-Export (eine Zeile pro Array-Item mit Wiederholung der Elternfelder). -- Server-seitige Aggregation (max. Array-Länge per Query) — kein passender Endpoint vorhanden. -- Styling/mehrere Sheets in XLSX — abgedeckt via `xlsx-js-style` (SheetJS-Fork mit `cell.s`-Styles); kein `exceljs` nötig. - ---- - -## 2. Ablauf (Flow) - -``` -[Export-Button in Toolbar] - │ - ▼ -[Export-Dialog: Format + Feldauswahl] - │ - ▼ -[Count-Check (getCount)] ──> wenn total > Schwellwert ──> [Count-Warnung (ConfirmDialog)] - │ - ▼ -[Batch-Laden aller Datensätze mit Fortschritts-Bar] - │ - ▼ -┌─ nur CSV/XLSX ─────────────────────────────┐ -│ [Post-Load-Check: max. Array-Länge je │ -│ "Dynamische Spalten"-Feld berechnen] │ -│ └─> wenn Spaltenzahl > Warnschwellwert ──>│ -│ [2. Dialog mit echten Zahlen] │ -└─────────────────────────────────────────────┘ - │ - ▼ -[Rows bauen + formatieren] ──> [Datei erzeugen] ──> [Download] -``` - -Der Export läuft **komplett clientseitig** über die bestehende `getEntries`-API. Kein neuer Server-Endpoint nötig. - ---- - -## 3. UI - -### 3.1 Button-Platzierung - -- In **Toolbar-Zeile 2** von `CollectionView.svelte` (die Zeile mit Filter / Sort / Bulk / View-Toggle / New Entry, ~Zeile 1976). -- Konsistent neben den anderen Listen-Aktionen, in **allen View-Modi** verfügbar (Tabelle, Cards, Media, Navigation) — Export betrifft die Daten, nicht die Darstellung. -- Öffnet den Export-Dialog (Modal, Muster wie `ConfirmDialog` / Create-Language-Dialog, mit `layerStack` für Z-Index). - -### 3.2 Export-Dialog (1. Dialog) - -**Format-Radio (3 Optionen):** -1. **JSON** — Default, sichere erste Wahl -2. **CSV** -3. **XLSX** - -**Feldauswahl:** -- Default = **aktuell sichtbare Tabellenspalten** (`tableColumns`). -- **„Alle auswählen"-Toggle** → alle geflatteneten Felder. -- Checkboxen pro Feld. - -**Format-abhängiges Verhalten:** - -| | JSON | CSV / XLSX | -|---|---|---| -| Feldauswahl-Granularität | Top-Level-Felder (rohe Struktur) | Geflattenete Blatt-Spalten | -| `object[]`-Strategie-Auswahl | **entfällt** | Strategie-Dropdown pro `object[]`-Feld | -| Post-Load-Check | **entfällt** (keine Spaltenexplosion) | ja (s. 3.3) | - -**`object[]`-Strategie-Dropdown (nur CSV/XLSX, nur wenn Feld angehakt):** -- **JSON pro Spalte** (Default — verlustfrei) -- **Dynamische Spalten** (`items.0.name`, `items.1.name`, …) - -### 3.3 Zweiter Dialog (nur CSV/XLSX, nach dem Laden) - -- Ausgelöst, wenn ein „Dynamische Spalten"-Feld mehr als den **Warnschwellwert** (Default 50) Spalten erzeugen würde. -- Zeigt **echte Zahlen** (max. Array-Länge × Subfeld-Anzahl = resultierende Spaltenzahl). -- **Kein hartes Limit** — „Trotzdem dynamisch" erzeugt die Spalten ohne Einschränkung. - -> „Feld `items` hat max. 120 Einträge → 120 × 3 Subfelder = **360 Spalten**. Auf JSON umschalten?" -> **[Auf JSON umschalten (empfohlen)]** **[Trotzdem dynamisch]** **[Abbrechen]** - ---- - -## 4. Daten laden: Batch + Fortschritt + Count-Warnung - -Da der Server `limit` begrenzen kann, **nicht** auf `_limit: 9999` verlassen, sondern **paginiert laden**: - -1. **`getCount(ns, col, filter)`** → Gesamtzahl (mit aktuellem Filterzustand). -2. **Count-Warnung:** wenn `total > THRESHOLD` (z.B. 500, analog `BULK_EDIT_COUNT_THRESHOLD`) → `ConfirmDialog` „Diese Sammlung enthält X Einträge. Export kann dauern. Fortfahren?". -3. **Batch-Loop:** `for offset = 0; offset < total; offset += BATCH` (BATCH = 500, konfigurierbar) → `getEntries({ _offset, _limit: BATCH, _select, _filter, _q, _lookup })`. - - Query-Parameter werden **beim Exportstart gesnapshotet** (Filter/Suchzustand einfrieren), damit Änderungen während des Ladens keine Auswirkung haben. - - **Fortschritt:** inline Progress-Bar im Dialog (`geladen / total`), aktualisiert pro Batch. -4. Nach dem letzten Batch: Rows bauen → Datei erzeugen → Download. - -**Fehlerbehandlung & Abbruch:** -- **Cancel-Button** im Dialog während des Ladens — bricht den laufenden Batch ab (via `AbortSignal`, das `getEntries` über `GetEntriesOptions.signal` unterstützt). -- **Batch-Fehler:** schlägt ein Batch fehl (Netzwerk/Timeout), wird er **einmal retried**; schlägt er erneut fehl, wird der Export abgebrochen, die Progress-Bar in einen Fehlerzustand versetzt und ein Fehler-Toast gezeigt. Bereits geladene Batches werden verworfen (kein teilweiser Export). - -**DRY:** Die Query-Bau-Logik (Filter + SubNav + `_q` + `_lookup`) aus `CollectionView` in eine geteilte Helper-Funktion `buildListQueryParams()` extrahieren — Liste & Export filtern damit identisch. - ---- - -## 5. Feld-Flattening (Objekte → Spalten) - -`flattenExportColumns(fields)` rekursiv durch `subFields`: - -- Jedes **Blatt** = eine Spalte, Pfad `parent.subfield` (z.B. `address.street`, `address.zip`). -- Header = `resolveLabel(field.meta.label)` (i18n) oder der Dotted-Path. -- Wert = `deepGet(record, path)` (existiert bereits in `preview-resolver.ts`). - -Damit ist `prop1.unterprop1`, `prop1.unterprop2`, … direkt gelöst: rekursives Flatten des Schemas. - -**Pfad → Feld-Auflösung für Formatierung:** `formatValue` benötigt ein `AdminCollectionField`-Objekt. Für geflattenete Dotted-Pfade (`address.street`) muss das Schema traversiert werden, um das SubField an diesem Pfad zu finden (analog `resolveFieldByPath` in preview-resolver.ts). Nur so funktionieren Datums-/Zahlen-/FK-Formatierung auch bei verschachtelten Feldern. - -**FK-Lookups für den geflatteneten Spaltensatz:** `getTableFkLookupEntries` arbeitet auf `TableColumn[]`. Für den Export muss der geflattenete Spaltensatz in eine parallele `TableColumn[]`-Struktur überführt (oder manuell Lookup-Specs für FK-Felder im geflatteneten Satz gebaut) werden, damit Fremdschlüssel als lesbare Werte exportiert werden. - -**i18n-Felder:** Felder mit `subFields` keyed by language code (i18n-Objekte) werden beim Flatten **nicht** in einzelne Sprach-Spalten aufgeteilt, sondern als **eine Spalte** mit dem Wert der aktuell gewählten Sprache exportiert (via `formatValue`/`resolveI18nObjectValue`). - ---- - -## 6. Array-Behandlung - -| Typ | Behandlung | -|-----|-----------| -| `string[]`, `number[]`, `file[]` | **Eine Spalte**, Werte mit **`,`** gejoint. Bei `file[]` die Pfade/URLs. | -| `object[]` | **Per-Feld konfigurierbar** (nur CSV/XLSX):
**(a) JSON pro Spalte** — das ganze Array als JSON-String (verlustfrei).
**(b) Dynamische Spalten** — `items.0.name`, `items.1.name`, … (nur 1. Ebene, tiefer = JSON). | -| `object[]` **innerhalb** eines `object[]`-Items | Komplett als JSON (bei Strategie „Dynamische Spalten"). | - -**Dynamische Spalten — Spaltenanzahl:** -- Da alle Datensätze ohnehin geladen werden, wird die **maximale Array-Länge über alle Datensätze** berechnet und daraus `items.0…items.(max-1)` erzeugt (datengetrieben, korrekt). -- Überschreitet die resultierende Spaltenzahl den **Warnschwellwert** (Default 50), greift der **zweite Dialog** (s. 3.3) — aber **ohne hartes Limit**. - -**Tradeoff (bewusst):** Dynamische Spalten machen den Export **nicht-deterministisch** — ändert sich die Datenlage (längeres Array), ändert sich die Spaltenanzahl zwischen zwei Exports. Für CSV/XLSX ok und erwartet. - ---- - -## 7. JSON-Export (Default) - -- Feldauswahl auf **Top-Level-Feldern** (rohe Struktur bleibt erhalten, verlustfrei). -- Export = **Array von Objekten** mit den ausgewählten Feldern. -- **Keine** `object[]`-Strategie-Auswahl und **kein** Post-Load-Check nötig. -- Batch-Laden + Fortschritt + Count-Warnung gelten trotzdem. - -**Format:** rohe (verschachtelte) Struktur der ausgewählten Top-Level-Felder — entspricht „einfacher JSON-Export". - ---- - -## 8. Packages - -**`xlsx-js-style`** (SheetJS-0.18.5-Fork mit Zell-Styles) als einziges neues Paket — deckt beide Tabellenformate ab: - -- `XLSX.utils.json_to_sheet(rows)` + `XLSX.writeFile(wb, "export.xlsx")` -- `XLSX.utils.sheet_to_csv(ws)` für CSV - -→ **DRY:** ein Paket, ein Code-Pfad, eine Row-Struktur. CSV-Quoting/Escaping übernimmt die Lib (Kommas in Werten werden korrekt gequotet). - ---- - -## 9. DRY / Wiederverwendung - -Bestehende Bausteine werden wiederverwendet statt neu gebaut: - -- **`deepGet`** (preview-resolver) → Wertextraktion -- **`formatValue`** (format-value.ts) → Datums-/Zahlen-/Bool-/FK-Formatierung (gleiche Darstellung wie in der Tabelle) -- **`resolveLabel`** → Spalten-Header (i18n) -- **`buildListQueryParams()`** (neu extrahiert aus `CollectionView`) → identische Filterung für Liste & Export -- **`getTableFkLookupEntries`** → FK-Lookups, damit Fremdschlüssel als lesbare Werte exportiert werden -- **UI-Pattern:** Dialog wie `ConfirmDialog`/Create-Language-Dialog, Toast-Feedback über `toast-store`, `layerStack` für Overlay-Z-Index -- **i18n:** neue Keys unter `admin.export.*` in `de.json`/`en.json` - ---- - -## 10. Neue Dateien & Änderungen - -### Neue Dateien - -1. **`frontend/src/lib/utils/export.ts`** - - `flattenExportColumns(fields)` → `ExportColumn[]` (Pfad, Label, Feld, Array-Strategie) - - `buildExportRows(records, columns)` → `(string | number)[][]` (mit `deepGet` + `formatValue` + Array-Join / JSON / Dynamische Spalten) - - `generateCsv(rows)` / `generateXlsx(rows, sheetName)` / `generateJson(records, fields)` + `triggerDownload(blob, filename)` - - `loadAllRecordsInBatches(params, onProgress)` → Batch-Loop mit Fortschritts-Callback - - `computeMaxArrayLengths(records, fields)` → für den Post-Load-Check - -2. **`frontend/src/lib/components/ui/ExportDialog.svelte`** - - Format-Radio (JSON / CSV / XLSX) - - Feld-Checkboxen (Default = sichtbare Spalten + „Alle"-Toggle) - - `object[]`-Strategie-Dropdown (nur CSV/XLSX) - - Progress-Bar - - Count-Warnung + zweiter Dialog (Post-Load-Check) - -### Änderungen - -3. **`CollectionView.svelte`** - - Export-Button in Toolbar-Zeile 2 - - Dialog-Verdrahtung - - `buildListQueryParams()`-Refactor (Query-Bau extrahieren) - -4. **i18n** — Keys unter `admin.export.*` in `de.json`/`en.json` - ---- - -## 11. Konfigurierbare Schwellwerte - -| Schwellwert | Default | Zweck | -|---|---|---| -| `BATCH_SIZE` | 500 | Batch-Größe beim Laden | -| `COUNT_WARNING_THRESHOLD` | 500 | Count-Warnung vor dem Laden | -| `COLUMN_WARNING_THRESHOLD` | 50 | Auslöser für den 2. Dialog (kein hartes Limit) | - ---- - -## 12. Entscheidungen (alle geklärt) - -- [x] **JSON-Export-Format:** **rohe (verschachtelte) Struktur** der ausgewählten Top-Level-Felder (verlustfrei, entspricht „einfacher JSON-Export"). -- [x] **Schwellwerte:** feste, sinnvolle Konstanten (Batch 500 / Count 500 / Spalten 50) — **nicht** im Dialog konfigurierbar. -- [x] **Einfache Arrays** (`string[]`, `number[]`, `file[]`): bleiben beim **Komma-Join** — kein Strategie-Dropdown dafür. Das `object[]`-Strategie-Dropdown existiert nur für `object[]`-Felder. - ---- - -## 13. Teststrategie (Playwright-only) - -**Wichtig:** Das Repo hat **keine Unit-Test-Infrastruktur** (kein Vitest/Jest) — nur Playwright (e2e / e2e-mobile / e2e-visual). Es wird **kein neues Test-Framework** eingeführt. Die pure Logik in `export.ts` wird über Playwright getestet, indem der **Download abgefangen und der Dateiinhalt validiert** wird: - -- **Download-Capture:** `page.waitForEvent('download')` → `download.saveAs(tempPath)` → Datei lesen. -- **CSV:** Text lesen, Zeilen parsen, Header/Quoting prüfen. -- **XLSX:** mit der `xlsx-js-style`-Lib im Test (`XLSX.read`, Styles via `{ cellStyles: true }`) das Sheet laden und Zellen prüfen. -- **JSON:** `JSON.parse` und Struktur/Felder prüfen. -- Die Tests laufen gegen den bestehenden **Mock-Server** (`frontend/src/lib/mock/`), mit geseedeten Fixtures (inkl. `object[]`-Feldern mit unterschiedlich langen Arrays). - -**Playwright-Testplan (UI-Flow + Download-Inhalt):** -- [ ] Export-Button in Toolbar sichtbar, öffnet Dialog. -- [ ] Format-Radio (JSON/CSV/XLSX), Default = JSON. -- [ ] Feldauswahl: Default = sichtbare Spalten, „Alle"-Toggle. -- [ ] `object[]`-Strategie-Dropdown nur im CSV/XLSX-Modus sichtbar. -- [ ] JSON-Export: alle gefilterten Datensätze, rohe Struktur, korrekte Feldauswahl (Download-Inhalt prüfen). -- [ ] CSV-Export: Header, Zeilen, FK-Werte, Quoting bei Kommas/Anführungszeichen (Download-Inhalt prüfen). -- [ ] XLSX-Export: Sheet-Zellen prüfen (Download-Inhalt prüfen). -- [ ] `object[]` Strategie „JSON pro Spalte" und „Dynamische Spalten" (Spaltennamen `items.0.name`, … prüfen). -- [ ] Dynamische Spalten mit max. Länge > Schwellwert → 2. Dialog, alle 3 Optionen. -- [ ] Batch-Laden: Fortschritts-Bar, korrekte Gesamtzahl, keine Duplikate/Lücken. -- [ ] Cancel-Button bricht Laden ab; Batch-Fehler → Retry, dann Abbruch + Fehler-Toast. -- [ ] Count-Warnung bei großer Sammlung (Abbrechen / Fortfahren). -- [ ] Filterzustand (Suchquery, Feld-Filter, SubNav) wird korrekt übernommen. - -**Hinweis:** Der Export respektiert den aktuellen Filterzustand. Soft-Delete-/`published`-Zustände werden **nicht** gesondert gefiltert — es wird exportiert, was die Liste zeigt (gleiche Semantik wie die Listenansicht). - ---- - -## 14. Schrittweiser Implementierungsplan - -> **Strenge Regeln für jeden Schritt:** -> 1. **Nach jedem Code-Change:** `LocalTester` mit dem engsten ausführbaren Check (`yarn validate` bzw. betroffene Playwright-Spec) **bevor** weitere Edits folgen. -> 2. **Nach jedem Code-Change:** `LocalReviewer` über den Diff — **vor** der finalen Antwort. -> 3. **Nach jedem Code-Change:** `index_repository` (Codebase-Memory) neu laufen lassen. -> 4. **Keine neuen Abhängigkeiten** außer `xlsx-js-style`. -> 5. **DRY:** bestehende Utilities wiederverwenden (`deepGet`, `formatValue`, `resolveLabel`, `layerStack`, `toast-store`, `ConfirmDialog`) — nichts duplizieren. -> 6. **Keine Server-Änderungen** — alles clientseitig. - -### Phase 0 — Vorbereitung -- [ ] `xlsx-js-style` als Dependency installieren (`yarn add xlsx-js-style`) -- [ ] i18n-Keys unter `admin.export.*` in `de.json`/`en.json` anlegen - -### Phase 1 — Pure Logik `frontend/src/lib/utils/export.ts` -- [ ] `flattenExportColumns(fields)` → rekursives Flatten, Dotted-Pfade, Header-Labels, i18n-Felder als eine Spalte -- [ ] Pfad→Feld-Auflösung (Dotted-Path → `AdminCollectionField`) für `formatValue` (analog `resolveFieldByPath`) -- [ ] `buildExportRows(records, columns)` → `deepGet` + `formatValue`, Komma-Join einfacher Arrays, `object[]`-Strategien (JSON pro Spalte / Dynamische Spalten), JSON-Fallback für tiefe Verschachtelung -- [ ] `computeMaxArrayLengths(records, fields)` → für Post-Load-Check -- [ ] `generateCsv` / `generateXlsx` / `generateJson` + `triggerDownload(blob, filename)` -- [ ] `loadAllRecordsInBatches(params, onProgress, signal)` → Batch-Loop (500), Retry (1×), AbortSignal -- [ ] FK-Lookup-Specs für den geflatteneten Spaltensatz bauen (Übersetzung von `TableColumn[]`) -- [ ] **Regel:** `yarn validate` grün - -### Phase 2 — Dialog `frontend/src/lib/components/ui/ExportDialog.svelte` -- [ ] Grundgerüst: Format-Radio (JSON/CSV/XLSX, Default JSON), Feld-Checkboxen (Default = `tableColumns`), „Alle auswählen"-Toggle -- [ ] `object[]`-Strategie-Dropdown (nur CSV/XLSX, nur für angehakte `object[]`-Felder) -- [ ] Progress-Bar + Cancel-Button -- [ ] Count-Warnung via `ConfirmDialog` (> 500) -- [ ] Post-Load-Check + 2. Dialog (Spaltenzahl > 50 → [Auf JSON umschalten] [Trotzdem dynamisch] [Abbrechen]) -- [ ] **Regel:** `yarn validate` grün - -### Phase 3 — Integration `CollectionView.svelte` -- [ ] `buildListQueryParams()` aus `CollectionView` extrahieren (DRY-Refactor, Filter + SubNav + `_q` + `_lookup`) -- [ ] Export-Button in Toolbar-Zeile 2 -- [ ] Dialog-Verdrahtung (State, Aufräumen bei Navigation/Component-Destroy) -- [ ] **Regel:** `yarn validate` grün - -### Phase 4 — Playwright-Tests (`tests/e2e/export.spec.ts`) -- [ ] Dialog-UI-Test (Button, Format-Radio, Default, Alle-Toggle, Strategie-Dropdown-Sichtbarkeit) -- [ ] JSON-Export-Test (Download-Inhalt: gefilterte Datensätze, rohe Struktur) -- [ ] CSV-Export-Test (Download-Inhalt: Header, Zeilen, FK-Werte, Quoting) -- [ ] XLSX-Export-Test (Download-Inhalt: Sheet-Zellen via `xlsx-js-style`-`read`, Header-Style prüfen) -- [ ] `object[]`-Strategie-Tests (JSON pro Spalte / Dynamische Spalten) -- [ ] 2. Dialog-Test (Spaltenzahl > Schwellwert, alle 3 Optionen) -- [ ] Count-Warnung-Test -- [ ] Cancel-/Retry-Test -- [ ] **Regel:** betroffene Playwright-Specs grün (`yarn test:e2e tests/e2e/export.spec.ts`) - -### Phase 5 — Abschluss -- [x] `LocalReviewer` über den kompletten Diff (Bugs, Regressions, Validierungslücken) -- [x] `yarn validate` + `yarn test:e2e` komplett grün -- [x] `index_repository` (Codebase-Memory-Sync) -- [x] Plan-Datei als „implementiert" markieren bzw. Erfahrungen zurückschreiben - -## 15. Umsetzungs-Notizen (Erfahrungen) - -- **Header für verschachtelte Felder:** Dotted-Path (`seo.metaTitle`) statt nur Leaf-Label — Leaf-Labels sind mehrdeutig (zwei Objekte können gleiche Subfeld-Labels haben). -- **`rows`-Feld in pages.yml hat Label „Seiteninhalt"** — Tests müssen Labels statt Feldnamen prüfen. -- **`content` (richtext) wird ausgeschlossen** — gleiche Semantik wie die Listenansicht (`getDisplayFields`). -- **Pagination-Bound:** `getCount` unterstützt kein `_q` (Suche) → der echte Gesamtwert kommt aus dem `count`-Header der ersten Batch-Antwort (`loadAllRecordsInBatches`). -- **CSV-Formula-Injection:** Zellen mit `=`, `+`, `-`, `@` werden mit `'`-Präfix sanitized (`sanitizeCsvCell`). -- **Abbrechen im Spalten-Warn-Dialog** schließt den Export-Dialog nicht (nur den aktuellen Export-Versuch) — UX-Fix aus Test EXP-10. -- **Dev-Server-Port:** `esbuild.config.js` liest jetzt `PORT`-Env (Default 3000) — nötig, weil Port 3000 in der Dev-Umgebung von openchamber belegt ist. Tests laufen mit `CODING_URL=http://localhost:3001`. -- **Vorbestehende Test-Failures:** `COLL-01`/`COLL-10` (collection-list) erwarten 20 Collections, Mock liefert 21 — Daten-Drift, unabhängig vom Export-Feature. diff --git a/src/content/docs/nova/custom-export-script.md b/src/content/docs/nova/custom-export-script.md deleted file mode 100644 index 62f6f86..0000000 --- a/src/content/docs/nova/custom-export-script.md +++ /dev/null @@ -1,121 +0,0 @@ ---- -title: 'Custom Export Scripts (`collection.meta.exports`)' ---- - -# Custom Export Scripts (`collection.meta.exports`) - -Custom exports are ES modules that run **client-side** in the ExportDialog. -One module may serve several exports — `ctx.exportName` tells them apart. - -## Config - -```yaml -# collections/.yml -meta: - exports: - - name: bestellreport - label: { de: "Bestellreport", en: "Order report" } - file: "_/assets/exports/reports.mjs" # _/... project-relative (recommended); - # also: /_/... (alias), /assets/... (host-absolute), https://..., data:, blob:, { eval } - filename: "bestellreport-{date}.xlsx" # optional, {date} → YYYY-MM-DD -``` - -No `select` — the script loads via `db.find` / `db.loadAll` itself (server syntax, see below). -Server schema: `tibi-types/schemas/config/collection-meta.schema.json` (`meta.exports`, -`additionalProperties: false` — `name` + `file` required). - -In the ExportDialog, collections with `meta.exports` show two tabs: **Predefined** -(custom scripts, active by default — no format/field selection) and **Generic** -(the standard JSON/CSV/XLSX export). Without `meta.exports` the dialog is -unchanged (generic only, no tabs). - -## Module contract - -```js -// _/assets/exports/reports.mjs -export default async function (ctx) { - if (ctx.exportName === "bestellreport") { /* … */ } - return { - workbook: ctx.xlsx.utils.book_new(), // OR blob | text | bytes (exactly one) - filename: "report-{date}.xlsx", // {date} supported - contentType: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", - } -} -``` - -Return: `{ filename, contentType }` + exactly one payload — -`workbook` (XLSX workbook, needs Excel contentType), `blob`, `text` (CSV/JSON/TXT), -`bytes` (`Uint8Array`/`ArrayBuffer`, e.g. PDF). Filenames are sanitized -(`/:*?"<>|` → `_`). See `frontend/src/lib/utils/custom-export-result.ts`. - -## `ctx` (mirrors tibi-server hook context where sensible) - -| Key | Notes | -|---|---| -| `api`, `project`, `collection` | GetMap-shaped JSON copies (safe to read, mutating is pointless) | -| `namespace`, `$project`, `$namespace`, `$method="export"`, `context` (self) | like `jsData` | -| `export` / `exportName` | own `meta.exports` entry / its `name` | -| `view` | current list state: `search`, `searchMode`, `sortField`, `sortOrder`, `manualSort`, `fieldFilters`, `subNavFilter`, `filter` (merged) | -| `db.find(coll, opts)` | server syntax: `{ filter, select, selector{…:1}, sort["-a","b"], offset, limit, lookup }`. Only `sort[0]` is used (REST limit). | -| `db.count(coll, { filter })` | server syntax | -| `db.loadAll(coll, opts, onProgress?)` | client helper: count → 500-batches (retry-once) + abort + progress. Or loop `find` manually. | -| `xlsx` | gebundelte `xlsx-js-style`-Lib (SheetJS + Zell-Styles, siehe unten) — kein Eigen-Import nötig | -| `helpers` | `deepGet`, `formatValue`, `csvCell` (formula-injection guard), `csvWithBom` (Excel umlauts) | -| `lang`, `languageCodes`, `defaultLanguage` | i18n | -| `progress(done, total, label?)` | dialog progress bar + label | -| `signal` | `AbortSignal` — "Abbrechen" button; `fetch` throws `AbortError` | - -No `$auth`: requests run as the logged-in admin (`fetchWithAuth`), the server -enforces permissions. Read-only: no create/update/delete. - -## Full example (multi-sheet Excel) - -```js -export default async function (ctx) { - ctx.progress(0, 0, "Lade Bestellungen…") - const orders = await ctx.db.loadAll( - "orders", - { select: ["orderNo", "totals"], filter: ctx.view.filter }, - (done, total) => ctx.progress(done, total, "Lade Bestellungen…") - ) - const wb = ctx.xlsx.utils.book_new() - ctx.xlsx.utils.book_append_sheet( - wb, - ctx.xlsx.utils.aoa_to_sheet([["Order", "Sum"], ...orders.map((o) => [o.orderNo, o.totals?.grandTotal ?? ""])]), - "Übersicht" - ) - const lines = orders.flatMap((o) => (o.items ?? []).map((it) => [o.orderNo, it.sku, it.qty])) - ctx.xlsx.utils.book_append_sheet(wb, ctx.xlsx.utils.aoa_to_sheet([["Order", "SKU", "Qty"], ...lines]), "Positionen") - return { workbook: wb, filename: "bestellreport-{date}.xlsx", contentType: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" } -} -``` - -Demo: `frontend/assets/mock/demo-exports/reports.mjs` (mock `pages` collection). -Types: `frontend/src/lib/utils/custom-export.types.ts` (single source of truth). - -## Styling (`cell.s` — nur via `xlsx-js-style` möglich) - -`ctx.xlsx` ist `xlsx-js-style` (SheetJS-API + schreibbare Zell-Styles). -Unterstützte `cell.s`-Keys: `fill`, `font`, `alignment`, `border`, `numFmt` -(Farben als ARGB, z.B. `"FF2F5496"`). - -```js -const HEADER_FILL = "FF2F5496" -function styleHeaderRow(ws, colCount) { - for (let c = 0; c < colCount; c++) { - const addr = ctx.xlsx.utils.encode_cell({ r: 0, c }) - if (ws[addr]) ws[addr].s = { - fill: { patternType: "solid", fgColor: { rgb: HEADER_FILL } }, - font: { bold: true, color: { rgb: "FFFFFFFF" } }, - } - } -} -const ws = ctx.xlsx.utils.aoa_to_sheet([["Order", "Sum"], ...orders.map((o) => [o.orderNo, o.totals?.grandTotal ?? ""])]) -styleHeaderRow(ws, 2) -``` - -Spaltenbreiten + Autofilter gehen ebenfalls ohne Zusatz-Lib: -`ws["!cols"] = [{ wch: 18 }, { wch: 14 }]` und -`ws["!autofilter"] = { ref: "A1:B10" }`. -Hinweis für Tests: `XLSX.read(buf, { cellStyles: true })` — ohne die Option -werden Styles beim Lesen ignoriert. diff --git a/src/content/docs/nova/konzept.md b/src/content/docs/nova/konzept.md deleted file mode 100644 index c4cfb11..0000000 --- a/src/content/docs/nova/konzept.md +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: Nova Konzept -description: Was Tibi-Admin-Nova ist und wie es betrieben wird. ---- - -**Tibi-Admin-Nova** ist das Svelte-5-Admin-Frontend für Tibi. - -- SPA als statisches Bundle, clientseitiges Routing, Daten über die Tibi-API. -- Lokal mit Mock-Modus entwickeln, in Staging/Live statisch hinter Reverse Proxy gegen den echten Tibi-Server. -- Laufzeitkonfiguration (nicht geheim) in `frontend/assets/system.config.json`, Proxy-Upstreams/Modelle in `config/ai-proxy.config.json`. -- Toolchain: `yarn install`, `make docker-start`, URLs unter `https://tibi-admin-nova.code.testversion.online/`. - -## Detailseiten - -Unter `nova/` liegen übernommene bzw. gepflegte Seiten: - -- `sammlungen.md` — Collection-Konfiguration im Admin -- `export.md` — Collection-Export -- `export-skript.md` — eigenes Export-Skript -- `testplaene/` — manuelle Testpläne - -## Konvention für Agenten - -- Neue Admin-Funktion → `tibi-admin-nova/docs/` aktualisieren, dann hierher synchronisieren (`npm run sync`), Screenshots bei UX-Änderungen. diff --git a/src/content/docs/nova/test-plans/02-navigation-routing.md b/src/content/docs/nova/test-plans/02-navigation-routing.md deleted file mode 100644 index 2a0e565..0000000 --- a/src/content/docs/nova/test-plans/02-navigation-routing.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: 'Navigation & Routing' ---- - -# Navigation & Routing - -## Scope - -- Authenticated SPA routing between project list, collection list, collection view, entry view, and action view. -- Browser history, breadcrumb navigation, and navigation-guard behavior. - -## Scenarios - -| ID | Scenario | Expected outcome | Priority | -| --- | --- | --- | --- | -| NAV-01 | Open /projects after auth | Project list renders without crashing | High | -| NAV-02 | Open /projects/:id | Collection list for the project renders | High | -| NAV-03 | Open /projects/:id/collections/:name | Collection view renders with entry actions | High | -| NAV-04 | Open /projects/:id/collections/:name/entries/:entryId | Entry view renders with editable shell | High | -| NAV-05 | Navigate via project card | SPA navigation happens without a full reload | High | -| NAV-07 | Use browser back after entering a project | Previous view is restored | High | -| NAV-08 | Use browser forward after going back | Forward history restores the collection list | Medium | -| NAV-09 | Open a deep collection URL directly | Correct route component loads | High | -| NAV-10 | Navigate to an unknown route | App shell stays usable and can recover | Medium | -| NAV-11 | Trigger navigation away from a dirty entry | Navigation guard blocks the route change | High | -| NAV-12 | Confirm discard in the guard dialog | Pending navigation continues | High | -| NAV-13 | Render breadcrumb on nested routes | Breadcrumb reflects the current hierarchy | Medium | -| NAV-14 | Click breadcrumb home | Navigation returns to the project list | High | -| NAV-14b | Click a parent breadcrumb segment from entry view | Navigation exits the entry route | High | -| NAV-14c | Click the collection breadcrumb from entry view | Navigation returns to the collection list and preserves remembered query state | High | - -## Regression Focus - -- Breadcrumb links from entry routes must use the same guarded SPA navigation path as other in-app links. -- The collection breadcrumb should reuse the remembered collection return URL when one exists so list filters and pagination survive the round-trip. \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/05-collection-view.md b/src/content/docs/nova/test-plans/05-collection-view.md deleted file mode 100644 index 6a40862..0000000 --- a/src/content/docs/nova/test-plans/05-collection-view.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -title: 'Test Plan: Collection View' ---- - -# Test Plan: Collection View - -## Scope - -- Collection list rendering for table, card, media, and navigation-capable collection views -- Shared collection search, sort, filter, pagination, selection, and URL-state behavior -- Entry navigation from the collection list, including actions triggered while debounced search is pending - -## Prerequisites - -- Playwright runs in mock mode against project `t1` -- Collections `pages`, `articles`, `blog`, `downloads`, `people`, and `products` are available in mock data -- The shared search input uses its standard debounce timing - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| CV-01 | Default collection view renders the table with rows and headers | Rows are visible and the list is interactive | High | -| CV-03 | View toggle switches between table and cards | The chosen view renders without a full-page reload | Medium | -| CV-04 | Search filters entries | URL updates with `q`, and only matching rows remain visible | High | -| CV-04b | Successive debounced searches abort stale in-flight list loads | Only the newest search request completes; earlier ones abort and do not overwrite results | High | -| CV-05 | Sorting updates the active order | Column sort changes the URL and list order state | Medium | -| CV-08 | Filter popover opens and applies field filters | Active filters narrow the result list and show visible state | Medium | -| CV-12 | Pagination appears for large collections and page changes work | The next page loads and URL state stays in sync | Medium | -| CV-18 | Clicking a row opens the entry editor | Navigation lands on the selected entry route | High | -| CV-19a | Opening a record during pending debounced search remains stable | Navigation to the entry editor is not reverted by a late search response | High | -| CV-19b | Creating a new record during pending debounced search remains stable | Navigation to `/entries/new` is not reverted by a late search response | High | -| CV-20 | Search state persists in the URL | Reloadable list state includes the active search term | Medium | - -## Edge Cases - -- Repeated typing across debounce boundaries must not allow stale search responses to flash old rows back into the table -- Silent background refetches must reuse the same latest-query rule and avoid overwriting newer list state -- Search-triggered navigation to entry routes must stay stable even if an older list request resolves later - -## Accessibility - -- The search input must remain keyboard-focusable and usable with shortcut-driven focus -- Table navigation and entry activation must continue to work after search, sort, and filter changes - -## Spec Mapping - -- `tests/e2e/collection-view.spec.ts` \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/07-entry-editor-advanced.md b/src/content/docs/nova/test-plans/07-entry-editor-advanced.md deleted file mode 100644 index feb0681..0000000 --- a/src/content/docs/nova/test-plans/07-entry-editor-advanced.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -title: 'Entry Editor Advanced — Test Plan' ---- - -# Entry Editor Advanced — Test Plan - -Tests: undo/redo, diff dialog, drilldown navigation, field search (Ctrl+F), jump targets (j chord). - -See `tests/e2e/entry-editor-advanced.spec.ts` for the Playwright implementation. - -## Field Search (Ctrl+F) - -| ID | Test Case | Steps | Expected Outcome | -| ----- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | -| EA-15 | Ctrl+F opens field search overlay | 1. Open an entry. 2. Press Ctrl+F. | Overlay with search input appears, focused. | -| EA-16 | Field search finds field by name | 1. Open field search. 2. Type a field name (e.g. "titel"). | Matches appear in the result list. | -| EA-17 | Field search navigates to field on selection | 1. Open field search. 2. Type field name. 3. Press Enter. | Overlay closes, target field is scrolled into view and flash-highlighted. | -| EA-20 | Field search expands accordion and navigates into flat `object[]` | 1. Open editor_lab/lab-base.
2. Open field search. 3. Search for unique value `#ffffff` (in `variants[0].variantValue`). 4. Press Enter. | Overlay closes. The collapsed `variants[0]` accordion item is expanded. The field `variants.0.variantName` is in the DOM. | -| EA-21 | Field search navigates through drilldown into accordion-mode `object[]` | 1. Open editor_lab/lab-base.
2. Open field search. 3. Search for "regionsname" (inside `campaign.channels[0].targeting.regions[0]`). 4. Press Enter. | Overlay closes. Breadcrumb shows "Kanaele #1". The `regions[0]` accordion is expanded, `regionName` field is visible. | - -### Bug Baseline (regression safeguard) - -Before the `expandAccordionAncestors` + `resolveFieldSegments` fixes: -- EA-20 failed: `object[]` with only primitive subfields was never expanded → field not in DOM. -- EA-21 failed: `object[]` inside drilldown added an extra drill segment for the flat array → landed one level too deep, empty content. - -## Jump Targets (j chord) - -| ID | Test Case | Steps | Expected Outcome | -| ----- | ---------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------- | -| EA-18 | `j` key opens jump list | 1. Open an entry. 2. Press `j`, wait 600ms. | Command palette opens, filtered to "Sektion". | -| EA-19 | `j` + digit jumps to section | 1. Open an entry. 2. Press `j` then `1` within 500ms. | Chord consumed, page scrolls to the first section. No overlay. | - -## Undo / Redo - -| ID | Test Case | Steps | Expected Outcome | -| ------ | --------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------- | -| EA-01 | Undo reverts last change | 1. Edit title. 2. Press Ctrl+Z. | Title reverts to original value. | -| EA-02 | Redo restores undone change | 1. Edit title. 2. Ctrl+Z. 3. Ctrl+Shift+Z. | Title re-applies the edit. | -| EA-03 | Undo availability indicator | 1. Edit title. 2. Check undo button. | Undo button becomes enabled. | -| EA-04 | Redo availability indicator | 1. Edit title. 2. Ctrl+Z. 3. Check redo button. | Redo button becomes enabled. | -| EA-05 | Undo button reverts last change | 1. Edit title. 2. Click undo button. | Title reverts. | -| EA-06 | Redo button restores undone change | 1. Edit title. 2. Undo. 3. Click redo button. | Title re-applies. | -| EA-07 | Multiple undos stepped | 1. Edit title twice. 2. Ctrl+Z twice. | Both changes reverted. | -| EA-08 | Undo after save preserves history | 1. Edit title. 2. Save. 3. Ctrl+Z. | Undo still works after save. | -| EA-09 | Undo updates modified badge | 1. Edit title. 2. Ctrl+Z. | Field "Geändert" badge disappears. | -| EA-10 | Dirty state toggles with undo/redo | 1. Edit title. 2. Ctrl+Z. 3. Ctrl+Shift+Z. | Dirty state reflects actual changes. | -| EA-11 | Redo button disabled after new edit | 1. Edit. 2. Undo. 3. Make new edit. | Redo button disabled (history branch). | -| EA-12 | Ctrl+Z at root closes drilldown first | 1. Drill into nested object. 2. Edit inside. 3. Ctrl+Z. | Undo inside drilldown works first. | -| EA-13 | Ctrl+Z at root closes drilldown after | 1. Drill and edit. 2. Ctrl+Z until drilldown pop. 3. Ctrl+Z at root. | Drilldown pops first, then undo at root. | -| EA-14 | Ctrl+Shift+Z redo after drilldown pop | 1. Drill, edit, pop. 2. Ctrl+Shift+Z. | Redo restores the drilled edit. | - -## Diff Dialog - -| EA-22 | Diff dialog shows field changes | 1. Edit several fields. 2. Click diff button. | Modal shows before/after comparison. | - -## Drilldown - -| EA-23 | Click drilldown card navigates nested | 1. Click a drilldown card. | Breadcrumb shows new level, sub-fields rendered. | -| EA-24 | Breadcrumb navigation pops level | 1. Drill in twice. 2. Click level 1 breadcrumb. | Returns to level 1, fields rendered. | -| EA-25 | ESC pops drilldown level | 1. Drill in. 2. Press ESC. | Returns to previous level. | diff --git a/src/content/docs/nova/test-plans/11-foreign-keys.md b/src/content/docs/nova/test-plans/11-foreign-keys.md deleted file mode 100644 index 1e713f2..0000000 --- a/src/content/docs/nova/test-plans/11-foreign-keys.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: 'Foreign Keys' ---- - -# Foreign Keys - -## Scope - -- Single `foreignKey` widgets in entry forms -- Multi-row `foreignKey` widgets and compact `ForeignKeyChipArrayWidget` chips -- Foreign key drawer browse/select flow and inline create flow -- Direct editing of already linked entries through nested drawers - -## Prerequisites - -- Run in mock mode only -- Use project `t1` -- Primary route: `/projects/t1/collections/blog/entries/cb1` -- Covered widgets: - - `author` for single FK editing/browsing - - `coauthors` for multi FK row rendering/browse flows - - `relatedArticles` for chip-array behavior - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| FK-01 | Single FK renders a resolved selected preview | The linked entry preview is visible and human-readable | High | -| FK-02 | Browse button opens the FK drawer | The drawer opens as a modal layer and lists target entries | High | -| FK-03 | Drawer search filters records | Result count changes and matching entries remain selectable | High | -| FK-04 | Selecting a record from the drawer updates the widget | The chosen record becomes the linked value and the single drawer closes | High | -| FK-05 | Removing a selected FK clears the value | The selected preview disappears and the widget returns to add/browse mode | High | -| FK-05b | Editing a selected single FK entry opens a nested edit drawer | The drawer shows the form plus a prominent referenced-by panel, saving refreshes the preview text | High | -| FK-05c | Object drilldowns inside the nested linked-entry drawer stay scoped to that drawer | Clicking a drilldown card renders breadcrumb plus nested fields inside the same edit drawer and persists nested edits | High | -| FK-06 | Multi FK widgets render multiple linked entries | Existing linked rows stay visible and removable/reorderable | High | -| FK-10 | Chip-array FK widgets render linked chips and support drawer pagination for large target collections | Chips stay interactive and the drawer pagination works across pages | Medium | -| FK-10a | Chip body tap/click opens the linked-entry editor | Saving updates the chip label while the chip remove button remains separate | High | -| FK-12 | Inline create is available from the FK drawer | The drawer can switch from browse mode into the embedded create form | High | -| FK-14d | Nested FK drawers keep browse shortcuts scoped to the topmost drawer | Sort/filter/view shortcuts in the inner FK drawer must not leak hidden state into the outer FK drawer's browse view | High | -| FK-15 | Create-and-select in single mode closes back to the field | The new entry is linked and visible in the widget preview | High | -| FK-16 | Create-and-select in multi mode keeps the drawer usable | The new entry is added without closing the overall FK drawer | High | -| FK-20 | Already selected records are highlighted in the drawer | Checked rows remain distinguishable and toggleable | Medium | -| FK-23 | FK drawers with sub-navigation expose category chips and respect them for inline create | The active filter drives create defaults and post-create visibility | Medium | - -## Edge Cases - -- The chip-array variant must not rely on hover; chip body opens edit, close button removes -- Direct linked-entry editing only targets already selected entries, not drawer search results -- The nested edit drawer must not replace the parent entry editor context or main route entry -- Object drilldowns inside the nested edit drawer must stay inside that drawer instead of hijacking the host editor -- Nested FK drawers must keep global browse shortcuts bound to the visible top drawer only -- Referenced-by information needs to stay visible high in the edit drawer, not buried below the form - -## Accessibility - -- Selected-entry edit actions remain keyboard reachable in list and card layouts -- Chip-array editing works on tap/click without hover-only affordances -- Nested edit drawers stay dismissible via ESC and reuse the existing drawer focus management \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/16-ai-integration.md b/src/content/docs/nova/test-plans/16-ai-integration.md deleted file mode 100644 index cace4eb..0000000 --- a/src/content/docs/nova/test-plans/16-ai-integration.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -title: 'Test Plan: AI Integration' ---- - -# Test Plan: AI Integration - -## Scope - -- AI chat panel open/close and message flow -- Tool-call recovery and retry behavior -- AI tool resolution for form-field choices and select inputs -- AI pagebuilder context for nested and flat block schemas -- Browser-native voice input in AI chat with API transcription as fallback - -## Core checks - -1. Open and close the AI chat via FAB and keyboard shortcuts. -2. Send a mocked chat message and verify the user turn appears in the panel. -3. Abort an active stream and verify the chat returns to an idle state. -4. Retry after a failed turn and preserve completed tool context. -5. For select/selectArray and foreignKey/foreignFile fields, AI tool input should resolve valid choice IDs, labels, and numeric 0-based indices to the actual stored choice ID, and reject out-of-range indices. -6. For pagebuilder context, `get_entry_context.pagebuilderFields` must expose the correct path shape for the active schema: nested paths like `rows.N.hero.title` only when the schema defines per-block objects, flat sibling paths like `rows.N.title` when fields are gated directly by `blockType` conditions. -7. For single-item pagebuilder arrays, FK/foreignFile drawer tools should auto-resolve omitted indices like `rows.mediaRef` to `rows.0.mediaRef` before opening/searching the drawer, so visible media options are not reported as empty. -8. When `SpeechRecognition` or `webkitSpeechRecognition` is available, start voice input through the browser API and do not call `/api/ai/transcribe`. -9. While native browser speech recognition is active, interim transcript updates should already appear in the chat textarea before stopping the recording. -10. After a longer pause without new browser speech-recognition events, the chat should auto-stop voice capture and keep the recognized text. -11. Overlapping browser speech-recognition result chunks must be merged without duplicating already recognized words in the final chat textarea, covering mobile Chrome style result streams. -12. When browser speech recognition is unavailable, keep the recorded-audio transcription fallback working via `/api/ai/transcribe`. -13. The microphone button should visualize the shared 3-second pause timeout by fading from active red back toward its normal state and reset to fully active when speech resumes. -14. Persisted AI conversation summaries must be merged into the single leading system message so follow-up turns never send a second `system` role later in the request. -15. On coarse-pointer mobile browser-STT, a short breathing pause must not end dictation at the desktop 3-second mark; the browser-specific pause grace should remain active until after that window. -16. If mobile browser-STT fires an unexpected browser `onend` before the pause timeout expires, dictation should resume automatically instead of finalizing the draft. -17. After voice capture enters pending-send, the UI must expose a visible cancel button for touch devices while keeping the ESC hint only for keyboard-capable devices; cancelling must preserve the text and prevent auto-send. -18. When the AI stream includes reasoning/thinking chunks before `delta.content`, the chat must show them in a subtle expandable panel during streaming and collapse that panel automatically once the final answer text arrives. -19. After `save_entry` creates a new record, the AI must land on the saved `/entries/{id}` route before a later `create_entry`; otherwise repeated create flows would keep editing the first record instead of opening a fresh form. -20. When a second `create_entry` opens directly from a just-saved entry, the `/entries/new` editor must be freshly mounted so prior widget-local UI state is gone and the form shows only defaults/empty values. -21. For date fields (type `date`), `fill_field` and `fill_form` must normalize flexible date inputs: date-only strings like `"2025-06-15"` → full ISO `"2025-06-15T00:00:00.000Z"`, full ISO strings pass through unchanged, and invalid date strings are passed through for server-side validation. -22. Recursive tool-call turns must be capped at a hard limit (25); exceeding it auto-stops with a user-friendly message. The counter resets on new user messages, new conversations, or final completion (no more tool calls). - -## Target specs - -- `tests/e2e/ai-chat.spec.ts` -- `tests/e2e/ai-tools.spec.ts` -- `tests/e2e/ai-advanced.spec.ts` -- `tests/e2e/ai-pagebuilder.spec.ts` \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/17-ai-tts.md b/src/content/docs/nova/test-plans/17-ai-tts.md deleted file mode 100644 index f97de6e..0000000 --- a/src/content/docs/nova/test-plans/17-ai-tts.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 'Test Plan: AI TTS' ---- - -# Test Plan: AI TTS - -## Scope - -- AI chat TTS for streamed assistant messages and manual playback -- Browser-native SpeechSynthesis as preferred engine -- OpenAI TTS API as fallback engine - -## Core checks - -1. Sentence splitting keeps incomplete trailing text until a real sentence boundary arrives. -2. Streaming chunks are batched into sensible TTS segments instead of per-token playback. -3. Parallel fetch completion order does not reorder spoken output. -4. TTS fetch errors are logged and do not crash the chat flow. -5. After cancel or a new prompt, later TTS playback still works. -6. When a matching local browser voice exists, the browser SpeechSynthesis engine is preferred and `/api/ai/tts` is not called. -7. When no matching local browser voice exists, playback falls back to `/api/ai/tts`. -8. When browser TTS is active with one matching browser voice, the chat should still show that browser voice in the menu together with the speed control. -9. When TTS falls back to the API, the chat should continue to offer the API voice selection list. -10. When multiple browser voices match the current frontend language, the chat should offer those browser voices and use the selected one for playback. - -## Target specs - -- `tests/e2e/ai-tts.spec.ts` \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/25-file-upload.md b/src/content/docs/nova/test-plans/25-file-upload.md deleted file mode 100644 index c4dcc88..0000000 --- a/src/content/docs/nova/test-plans/25-file-upload.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -title: 'File Upload' ---- - -# File Upload - -## Scope - -- FileWidget uploads in entry editors -- Drag-and-drop, preview, image editor and downscale behavior -- Automatic media AI autofill after direct image selection in entry view - -## Prerequisites - -- Run in mock mode only -- Use project t1 -- Fake every `/api/ai/vision` call inside the test; never depend on a live AI endpoint -- Collections: - - editor_lab for generic file/image widget coverage - - media_library for image downscale, palette extraction and media AI autofill - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| F-01 | Upload image via FileWidget | Preview and metadata appear after selection | High | -| F-12 | Large image upload triggers configured downscale | Notice reflects resized dimensions and preview remains usable | High | -| F-15e | Direct media file selection triggers media AI autofill | The app posts a data URI to the vision endpoint and fills the configured target field | High | -| F-15f | Media AI autofill does not overwrite existing target content | Already filled target values stay unchanged; only missing values are written | High | -| F-15g | Media AI pending state is visible and cancelable | Inline AI status appears on the file widget and cancel prevents autofill | High | -| F-15h | Manual media AI rerun overwrites existing image descriptions | Existing images expose a manual AI trigger in the action slot and replace already filled target values | High | -| F-16 | Duplicate an entry with direct file fields | The new draft keeps normal field values, clears direct file/file[] values, and shows a hint that uploads must be repeated | High | - -## Edge Cases - -- Vision request uses a resized data URI when media AI image sizing is configured -- Duplicating an entry must not carry direct file objects over to the new entry unless a real server-side copy flow exists -- i18n object targets receive structured JSON and fill only missing language keys -- If the AI endpoint fails, the file upload still succeeds and the widget keeps the selected file -- Canceling the inline media AI run keeps the selected file and leaves target fields untouched -- Existing images can rerun media AI manually from the action slot and intentionally overwrite current target values - -## Accessibility - -- File input remains keyboard reachable through the widget trigger -- i18n language tabs remain operable after automatic autofill -- AI autofill must not block normal save or focus flow in the entry editor \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/27-media-grid.md b/src/content/docs/nova/test-plans/27-media-grid.md deleted file mode 100644 index 47649ec..0000000 --- a/src/content/docs/nova/test-plans/27-media-grid.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -title: 'Media Grid Test Plan' ---- - -# Media Grid Test Plan - -## Scope - -- Media grid rendering and view switching for collections with `meta.viewHint.media` -- Upload queue behavior, including `Edit & upload` and inline editor drawer flows -- Regression coverage for i18n-aware fields rendered inside the shared upload-queue `InlineEntryForm` - -## Prerequisites - -- Mock mode with project `t1` -- Collection `media_library` available under `/projects/t1/collections/media_library` -- Playwright spec: `tests/e2e/media-grid.spec.ts` - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| MG-01 | Media grid opens as default view for `media_library` | Grid is visible and URL resolves to `?view=media` | High | -| MG-05b | Wechsel von einer anderen Collection nach `media_library` setzt den Media-Default erneut | Ein zuvor gewaehlter Tabellen-/Kartenmodus aus der Quell-Collection wird nicht uebernommen; `media_library` oeffnet wieder im Grid mit `?view=media` | High | -| MG-11a | Upload queue `Edit & upload` opens the inline editor drawer | A dialog with the shared `InlineEntryForm` opens | High | -| MG-11b | Inline editor shows media collection fields | Editable collection fields render; auto-filled non-editable metadata stays hidden | High | -| MG-11e | Inline editor renders i18n-shaped fields with language tabs, toolbar switcher, and `Ctrl+L` cycling | The drawer shows the global language toggle plus per-field i18n tabs; clicking the toolbar toggle or pressing `Ctrl+L` updates the active field tab instead of exposing raw `de`/`en` object editors | High | -| MG-16 | Upload queue titles use filename-derived auto-fill | Queue rows show the normalized filename title without extension | Medium | -| MG-18 | Quick upload waits for the current queue item's media AI result | The create request is sent only after the item's AI caption is ready, and the request body already contains the generated i18n caption | High | -| MG-19 | Earlier queue items can upload while later queue items still run media AI | The first finished item uploads immediately while later rows keep their running AI status until their own caption is ready | High | -| MG-20 | Stopping a running queue AI task pauses it without surfacing an error and allows a manual restart | After pressing stop the row shows a restart action instead of the error chip, and restarting completes the upload with the generated caption | High | - -## Edge Cases - -- Multiple queued files keep independent pending states while one drawer is open -- Upload drawer keeps i18n state scoped to the current inline form instance -- Optional i18n fields remain optional and must not block upload when left empty -- All media AI requests are faked in Playwright; upload-queue coverage must not depend on a live `/api/ai/vision` backend - -## Accessibility - -- Upload queue actions remain keyboard reachable -- The inline editor drawer exposes a proper dialog role and focus management -- Language toggles and i18n tabs stay operable via keyboard in the drawer \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/34-i18n-fields.md b/src/content/docs/nova/test-plans/34-i18n-fields.md deleted file mode 100644 index 804a562..0000000 --- a/src/content/docs/nova/test-plans/34-i18n-fields.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -title: '34 i18n Fields' ---- - -# 34 i18n Fields - -## Scope - -Field-level i18n editing in project t1, including language tabs, active-language switching, and AI-assisted subtree translation for supported text widgets. - -## Prerequisites - -- Mock mode enabled -- Project t1 -- Collection i18n_content -- Entry ic2 for missing EN translations - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| I18N-01 | i18n field shows language tabs | Tabs for configured languages are visible on detected i18n fields | High | -| I18N-10 | Switching tabs changes visible content | Active language content changes without losing the other language value | High | -| I18N-30 | Editing one language marks the entry dirty | Unsaved badge appears after editing an i18n value | High | -| I18N-33 | AI translation fills a missing simple target value | Switching to the target tab and clicking translate inserts the translated target value and marks the entry dirty | High | -| I18N-34 | AI translation fills a nested subtree | Switching to the target tab and clicking translate on a nested i18n object fills supported target leaves while preserving the field shape | High | - -## Edge Cases - -- Translation button is hidden for readonly i18n fields -- Empty source language content does not overwrite the target -- Invalid AI JSON response shows an error toast and does not mutate the field -- Only supported string widgets are translated; other values remain local copies from the source subtree - -## Accessibility - -- Language tabs remain keyboard-focusable buttons with correct tab semantics -- Translate actions expose a descriptive aria-label for translating into the active target language from the auto-selected best source language \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/35-entry-i18n.md b/src/content/docs/nova/test-plans/35-entry-i18n.md deleted file mode 100644 index 0c56aae..0000000 --- a/src/content/docs/nova/test-plans/35-entry-i18n.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: 'Entry I18n' ---- - -# Entry I18n - -## Scope - -- Entry-level i18n collections in project t1 -- Create flow for collections that store one language per entry -- Draft language selection before the first save - -## Prerequisites - -- Run in mock mode only -- Use project t1 -- Collections: - - articles_i18n for default-language create flow - - articles_i18n_picklang for explicit language picker flow - - articles_i18n_nested for nested languageField/groupField tests - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| EI18N-06 | Open /entries/new for articles_i18n | Draft language switcher is visible, default language is preselected, hidden system fields stay hidden | High | -| EI18N-07 | Change draft language before first save | Selected draft language becomes active and remains active after save | High | -| EI18N-08 | Create from collection without usable default language | Language picker appears before navigation and selected language is prefilled in the draft | High | -| EI18N-08b | Duplicate an existing translated entry from EntryView | New draft keeps the source language active and visible translatable fields are prefilled from the source entry | High | -| EI18N-09 | Run AI translate on an existing target-language entry | Best sibling source is picked by content score and language order, translatable fields update, non-translatable fields stay unchanged | High | -| EI18N-24 | Use the Nova icon inside the create-language menu | The normal row still creates a copy, the Nova action creates the target language and immediately starts translation from the currently open source entry | High | -| EI18N-13 | Edit a normal field in an existing translated entry | TranslationBar state stays stable and no extra sibling list GET is triggered while typing | High | -| EI18N-12 | Press Ctrl+L on an existing translated entry | Entry view navigates to the next sibling language instead of only flashing the shortcut highlight | High | -| EI18N-25 | Create translation from existing entry navigates to create mode with pre-filled fields | After clicking a target language in the create menu, the browser navigates to /entries/new?lang=; required fields from the source entry are preserved, non-required clearFields are emptied | High | -| EI18N-26 | Create translation with unsaved changes shows dirty guard first | If the source entry has unsaved changes, a discard dialog appears before navigating to create mode | Medium | -| EI18N-60 | Nested i18n: details.lang and details.translationGroup are hidden from form | The object field "details" is visible but the sub-fields "lang" and "translationGroup" are not rendered | High | -| EI18N-61 | Nested i18n: regular fields are visible | title, slug, summary, body are visible and editable | High | -| EI18N-62 | Nested i18n: TranslationBar shows language tabs | TranslationBar renders with DE active, EN/FR siblings visible | High | -| EI18N-63 | Nested i18n: create translation from existing entry | Creating a translation navigates to create mode with pre-filled fields, details.lang is set to the target language | High | - -## Edge Cases - -- Direct deeplink to /entries/new should still get a usable default if one exists -- Collections without a default language must not silently create the wrong language -- Duplicating an EN or FR entry must not reopen the draft in the collection default language and hide the copied values in another language tab -- When multiple sibling source entries have equal content, the earlier configured language should win as deterministic fallback -- The Nova action in the create menu must keep the source pinned to the currently open language entry, even if another sibling would score higher globally -- Ctrl+L must not be swallowed by field-i18n logic when the collection only uses entry-level i18n -- Unsaved typing in non-language fields must not re-request sibling translations for the current translation group -- Nested i18n config (languageField inside an object field): the parent object stays visible but system sub-fields (lang, translationGroup) are filtered out in FieldRenderer -- Creating a translation in nested config: clone must correctly handle dotted clearField paths (e.g. empty nested title/body) -- Unterbrochener Create-Mode (Benutzer navigiert weg vor dem Speichern) muss keine verwaisten POSTs hinterlassen — der create-mode Clone-Flow erzeugt erst beim expliziten Speichern einen POST - -## Accessibility - -- Language picker must be a modal dialog with focus trap and ESC close -- Draft language controls must stay keyboard accessible -- Existing-entry language switching via Ctrl+L must remain keyboard accessible and trigger real navigation \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/36-voice-input-fields.md b/src/content/docs/nova/test-plans/36-voice-input-fields.md deleted file mode 100644 index 0d624df..0000000 --- a/src/content/docs/nova/test-plans/36-voice-input-fields.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: 'Test Plan: Voice Input Fields' ---- - -# Test Plan: Voice Input Fields - -## Scope - -- Shared STT behavior for plain-text admin inputs -- Search inputs using the shared search component -- Standard single-line and multiline text widgets in entry forms -- Richtext widgets in WYSIWYG and source-view mode -- Foreign-key inline search fields used in filters and forms -- Free-text inputs inside the bulk edit panel - -## Core checks - -1. Mic control for plain-text fields appears only when the field is focused, unless recording/transcribing is active. -2. Search inputs can be filled through browser voice input and still apply their normal debounced filtering behavior. -3. Text widgets can be filled through browser voice input and keep the recognized text editable afterwards. -4. Richtext widgets can insert recognized text at the current ProseMirror selection. -5. Richtext source mode can be filled through the same focused mic control pattern as plain-text textareas. -6. Foreign-key inline search fields can be filled through voice input and keep their dropdown search behavior intact. -7. Foreign-key filter fields, including multi-select `string[]` variants, expose the same right-aligned browse button as the regular FK widget. -8. Voice errors surface as toasts instead of chat-only inline messages. -9. Bulk-edit free-text inputs, including string operation values and search/replace text fields, can be filled through the shared mic control without breaking the panel flow. -10. Leaving a debounced search input and immediately triggering route navigation (open record / new record) must not bounce the user back to the collection view. - -## Target specs - -- `tests/e2e/collection-list.spec.ts` -- `tests/e2e/collection-view.spec.ts` -- `tests/e2e/foreign-key-filter.spec.ts` -- `tests/e2e/widgets.spec.ts` -- `tests/e2e/bulk-operations.spec.ts` \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/37-pagebuilder.md b/src/content/docs/nova/test-plans/37-pagebuilder.md deleted file mode 100644 index b4ae67a..0000000 --- a/src/content/docs/nova/test-plans/37-pagebuilder.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -title: 'Pagebuilder' ---- - -# Pagebuilder - -## Scope - -- Pagebuilder thumbnail and live-preview rendering in project t1 -- FK lookup hydration for block registries and mock/demo blocks -- URL normalization for linked media in pagebuilder rows - -## Prerequisites - -- Run in mock mode only -- Use project t1 -- Relevant spec: tests/e2e/pagebuilder.spec.ts - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| PB-32 | Hero preview without explicit bgImage uses linked mediaRef lookup | Thumbnail/live preview renders the linked media image instead of an empty background | High | -| PB-36b | Direct image-widget block in editor_lab pageContent | Before save, the live thumbnail renders the direct block image source in live mode and the mock placeholder variant in mock mode; after save, the generated screenshot replaces the raw image source | High | -| PB-40b | Lookup normalization receives a data: or blob: media URL | Inline media URI stays unchanged and is not prefixed with /api/_// | High | -| PB-40c | Existing server-style nested `_lookup` entries in pagebuilder rows are hydrated for preview | Pagebuilder hydration rewrites nested lookup `file.src` values to absolute admin URLs without changing the underlying mock/API payload shape | High | -| PB-35b | Nested object[] items inside a pagebuilder block use FK/foreignFile lookups | Thumbnail/live preview resolves indexed paths like slides.0.image into the correct child item instead of leaving nested media unresolved | High | - -## Edge Cases - -- Newly uploaded media may exist as data URI before backend persistence; pagebuilder previews must still render it. -- Direct file/image widgets inside screenshot-enabled pagebuilder blocks must be included in both live thumbnail rendering and saved screenshot generation. -- Blob URLs from client-side editors or upload flows must remain browser-local and must not be rewritten. -- Lookup embedding must honor numeric path segments for nested object[] descendants inside pagebuilder blocks. -- Existing server-style `_lookup` payloads inside pagebuilder rows must stay server-style in mock/API data, but preview hydration still has to absolutize nested `file.src` values for admin rendering. - -## Accessibility - -- Thumbnail previews remain decorative only; no additional focus targets are introduced by the media rendering path. \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/38-foreign-file.md b/src/content/docs/nova/test-plans/38-foreign-file.md deleted file mode 100644 index 8a42361..0000000 --- a/src/content/docs/nova/test-plans/38-foreign-file.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -title: 'ForeignFile / ForeignMedia' ---- - -# ForeignFile / ForeignMedia - -## Scope - -- Single and multi `foreignFile` / `foreignMedia` widgets in the admin editor -- Direct upload via the shared `FileUploadQueue` -- Shared media-AI queue behavior for direct uploads into `media_library` -- Direct editing of already linked media entries via nested drawers - -## Prerequisites - -- Run in mock mode only -- Use project `t1` -- Fake every `/api/ai/vision` call inside the test; never depend on a live AI endpoint -- Entry route: `/projects/t1/collections/blog/entries/cb1` -- Widget targets: - - `headerImage` for single foreign media - - `relatedMedia` for multi foreign media - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| FF-07a | Mock-mode thumbnails for preselected related media stay browser-loadable | Existing `cm1`/`cm2`/`cm3` image cards render mock-safe thumbnail URLs instead of broken `/api/_/.../file/...` requests | High | -| FF-05b | Editing the selected single media entry opens the nested edit drawer | The drawer shows media fields plus a prominent referenced-by panel, saving refreshes the widget preview | High | -| FF-05c | Object drilldowns inside the nested media edit drawer stay scoped to that drawer | Clicking a drilldown card renders breadcrumb plus nested fields inside the same edit drawer and persists nested edits | High | -| FF-10a | Editing a selected media card in multi mode updates only that linked card | The card title/preview refreshes in place and the rest of the selection stays intact | High | -| FF-12 | Uploading a file opens the shared queue/editor flow in single mode | The inline editor opens and the queue item remains editable/cancelable | High | -| FF-17 | Single-mode quick upload creates and selects the new media entry | The widget immediately resolves the created media entry as the selected value | High | -| FF-17b | Single-mode quick upload waits for media AI before create | The create payload contains the generated i18n caption and starts only after the item's AI finished | High | -| FF-17c | Multi-mode quick upload appends a newly uploaded media entry | The new created media entry is added to the existing selection without replacing older items | High | -| FF-17d | Multi-mode quick upload all respects per-item AI completion | Earlier queue items upload as soon as their own AI is ready while later items may still show running AI status | High | -| FF-17e | Nested drilldown foreign media queues survive leaving the drilldown | A prepared queue item is preserved when the drilldown closes and is still flushed before entry save | High | - -## Edge Cases - -- Single mode hides the upload zone while a queue item is pending; users operate on the queue entry instead -- Multi mode keeps the upload zone visible while selected entries and queue items coexist -- Selected media cards expose their edit action without requiring hover so touch devices can reach it -- Nested object drilldowns inside the media edit drawer must stay scoped to that drawer instead of the host editor -- The shared queue behavior must match the `viewHint.media` upload queue semantics for AI timing and payload content -- Drilldown-mounted foreign media widgets must preserve prepared queue items across unmount/remount and still flush them on save - -## Accessibility - -- Upload zone actions remain keyboard reachable in both single and multi mode -- Inline editor dialog remains dismissible via keyboard after adding queued files -- Direct upload controls stay operable without relying on hover-only affordances -- Linked media edit actions stay visible and keyboard reachable in both single and multi card layouts \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/39-preview-resolver.md b/src/content/docs/nova/test-plans/39-preview-resolver.md deleted file mode 100644 index 3422597..0000000 --- a/src/content/docs/nova/test-plans/39-preview-resolver.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 'Preview Resolver' ---- - -# Preview Resolver - -Scope: Preview semantics in t1 for collection cards/tables, FK mini previews, and object or object[] previews inside entries. - -Fixtures: -- Project: t1 -- Entry: /projects/t1/collections/editor_lab/entries/lab-base -- Collections: blog, people, downloads, pages, media_library, editor_lab - -Regression focus: -- Without explicit preview config, preview auto-detection must still pick meaningful visible fields. -- With explicit preview config, preview stays strict and must not auto-fill missing title, meta, image, or extra object fields. -- Explicit slot paths in dot notation such as profile.name or profile.photo must resolve in collection cards, tables, media previews, and FK-driven target previews. - -Cases: -- PR-01 to PR-17: Existing coverage for FK fallback, collection tables/cards, eval previews, and object[] eval labels. -- PR-18: readonly object[] without meta.preview still shows meaningful auto-detected values instead of collapsing to ids or empty output. -- PR-19: object[] with explicit meta.preview eval shows only configured preview text and does not append other values such as variantValue or boolean fields. -- PR-20: people collection uses nested preview slot paths profile.name, profile.position, and profile.photo for card/table rendering and related FK previews. -- PR-21: blog coauthors FK chips resolve people slot previews with label plus secondary text and do not fall back to raw ids. -- PR-22: list/card badge slots that point to foreign keys resolve the target collection preview text instead of showing the stored FK id. -- PR-23: structured FK previews keep badge slots foreign-aware as well, so chips and inline mini previews do not regress to raw ids. -- PR-24: collection tables keep foreign-key cell previews foreign-aware even when the target preview badge itself points to another foreign key. -- PR-25: FK drawer tables keep the same nested badge resolution, so browse rows do not regress to raw foreign ids. \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/40-actions.md b/src/content/docs/nova/test-plans/40-actions.md deleted file mode 100644 index 9f98462..0000000 --- a/src/content/docs/nova/test-plans/40-actions.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -title: 'Test Plan: Actions' ---- - -# Test Plan: Actions - -## Scope - -- Project action discovery in the project overview -- Action detail routing and method-tab state -- Runnable vs. locked action methods -- Action execution result and progress rendering - -## Core checks - -1. Action cards in the project overview open the expected action detail route. -2. Method badges and method tabs keep the selected method in the query string while the action route is active. -3. Leaving the action detail route after switching methods must not carry the `method` query parameter onto the next route. -4. Back navigation from an action detail returns to the project overview without reopening the action. -5. Runnable actions show inputs, can execute, and render output or progress consistently. -6. Locked or non-runnable actions expose the correct disabled state and hint messaging. - -## Target specs - -- `tests/e2e/actions.spec.ts` \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/41-search-index-status.md b/src/content/docs/nova/test-plans/41-search-index-status.md deleted file mode 100644 index a809f2d..0000000 --- a/src/content/docs/nova/test-plans/41-search-index-status.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -title: 'Test Plan: Search Index Status Notice' ---- - -# Test Plan: Search Index Status Notice - -## Scope - -- Project-level search-index status notice in the collection overview header -- Collection-level search-index status notice in the collection header -- Live polling and automatic notice removal when the queue drains - -## Mock anchor - -- Mock project `t1`, collection `products` -- `products` exposes one `ngram` search mode and two records with `_queuePending: true` - -## Core checks - -1. The project overview aggregates pending `_queue` counts across collections that expose `ngram` or `vector` search modes. -2. The collection header shows the notice only for the current collection when its `_queue` count is greater than zero. -3. The notice disappears automatically after the next polling cycle once `_queue` returns zero. -4. Collections without relevant search modes do not trigger the queue check and do not show the notice. - -## Target specs - -- `tests/e2e/project-list.spec.ts` -- `tests/e2e/collection-view.spec.ts` \ No newline at end of file diff --git a/src/content/docs/nova/test-plans/42-impersonation.md b/src/content/docs/nova/test-plans/42-impersonation.md deleted file mode 100644 index fc937f1..0000000 --- a/src/content/docs/nova/test-plans/42-impersonation.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: 'Impersonation (Admin → User Session)' ---- - -# Impersonation (Admin → User Session) - -## Scope - -- Impersonation einer Benutzersession durch einen Admin -- `POST /login/impersonate` Endpoint im Mock-Interceptor -- UI-Trigger im UserDetailDrawer (Button + ConfirmDialog) -- Amber-Banner im AdminLayout mit Stop-Button -- Rollback zur Admin-Session via `stopImpersonation()` -- Berechtigungen: SystemAdmin (role=0) kann alle User impersonaten; OrgAdmin nur User in eigenen Orgs (Serverseitig geprüft) - -## Prerequisites - -- Run in mock mode only (real impersonation requires live tibi-server) -- Admin-Benutzer `admin` (role=0) und Ziel-User `maria.schmidt` (u2) existieren -- Zweiter Admin `peter.gross` (u7) existiert für Admin-auf-Admin-Tests -- Relevant spec: `tests/e2e/impersonation.spec.ts` - -## Test Cases - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| IMP-01 | Admin sieht "Als Benutzer einloggen"-Button im UserDetailDrawer | Button mit `mdiLogin` Icon ist sichtbar für role=0 | High | -| IMP-02 | Admin klickt Impersonate → ConfirmDialog erscheint | ConfirmDialog variant="info" mit Titel und Nachricht | High | -| IMP-03 | Admin bestätigt Impersonation → Weiterleitung zu /projects | Amber-Banner "Eingeloggt als @maria.schmidt" sichtbar; User-Menü zeigt Maria Schmidt | High | -| IMP-04 | Impersonation-Banner zeigt korrekte Impersonation-Info | Banner enthält "(Impersonation SystemAdmin)" | High | -| IMP-05 | Stop-Button im Banner → Admin-Session wiederhergestellt | Zurück zu /projects mit Admin-User im User-Menü; Banner verschwunden | High | -| IMP-06 | Nicht-Admin (role=1) sieht KEINEN Impersonate-Button | Button ist nicht sichtbar für maria.schmidt | High | -| IMP-07 | Admin sieht Impersonate-Button NICHT für sich selbst | admin (u1) kann sich nicht selbst impersonaten | Medium | -| IMP-08 | Admin-zu-Admin Impersonation (admin → peter.gross) funktioniert | Impersonation auf peter.gross möglich; Banner zeigt "@peter.gross" | Medium | -| IMP-09 | Impersonation-Banner verschwindet nach page.reload() | sessionStorage stellt impersonatedBy-State wieder her | Medium | -| IMP-10 | Admin sieht Impersonate-Button auch für User ohne Org/Team | solo.user (u8) hat Impersonate-Button | Low | - -## Edge Cases - -- Impersonation des eigenen Users: Button muss versteckt sein (canImpersonate = isAdmin && user.id !== currentUser.id) -- Mock-Interceptor lehnt Impersonation ab, wenn currentUser nicht role=0 ist (403) -- Mock-Interceptor gibt 404 bei unbekannter userId -- Nach Stop-Impersonation muss die Admin-Original-Session wiederhergestellt sein (via singleRefresh) -- Bei fehlendem Original-Token (sessionStorage gelöscht) → full logout (Fallback) - -## Accessibility - -- Amber-Banner hat `data-ai-id="impersonation-banner"` für Test-Selektion -- Stop-Button hat `data-ai-id="impersonation-stop-btn"` -- Impersonate-Button hat `data-ai-id="impersonate-btn"` -- Farbkontrast: Amber auf hell/dunkel erfüllt WCAG AA diff --git a/src/content/docs/nova/test-plans/43-backup-restore.md b/src/content/docs/nova/test-plans/43-backup-restore.md deleted file mode 100644 index 3f141ce..0000000 --- a/src/content/docs/nova/test-plans/43-backup-restore.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: 'Backup & Restore' ---- - -# Backup & Restore - -## Scope - -- Admin UI: Backup export (SSE progress, download) and import (upload, analyze, restore) -- API communication via `backup-api.ts` (SSE parser, 6 endpoints) -- Mock interceptor coverage for all backup routes -- Access control: project admin permission required - -## Prerequisites - -- Run in mock mode (default) -- Use project t1 -- Admin token must have `AllowBackup: true` - -## Test Cases - -### Export (Create Backup) - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| BCK-01 | Admin menu "Backup & Restore" button visible on project page | Button appears in admin dropdown; clicking navigates to `/projects/:id/backup` | High | -| BCK-02 | Admin menu opens dropdown with backup entry and navigates | Dropdown shows "Backup & Restore" link; clicking navigates to backup route | High | -| BCK-03 | Direct deep-link to backup route renders | Visiting `/projects/:id/backup` shows the backup page with export tab active | High | -| BCK-04 | Export tab shows collections with header checkbox and pre-selected state | Table lists all non-hidden collections; header checkbox is checked (all selected) | High | -| BCK-05 | Backup export SSE flow completes | Clicking "Start backup" shows progress section; SSE events arrive; final done state shows download button with size and countdown | High | -| BCK-06 | Import tab shows upload zone | Switching to import tab shows drag-and-drop zone for ZIP files | High | -| BCK-07 | Backup back button navigates to project | Clicking back arrow returns to project page | High | -| BCK-08 | Existing action routes still work (no regression) | Other admin routes (e.g. search, inspect) remain functional after backup route registration | High | -| BCK-09 | Unknown action deep-link still shows not-found (no regression) | Visiting `/projects/:id/backup/nonexistent` shows no route match | High | - -### Export — Analyze-Download Estimates - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| BCK-10 | Export table shows media file counts from analyze-download | Each collection row has media count and media size; grand summary shown below table | Medium | -| BCK-11 | Analyze-download handles server error gracefully | If endpoint fails, table shows "—" placeholders; no crash | Medium | - -### Export — Progress Display - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| BCK-12 | Step progress bar shows per-collection documents/media | During backup, blue step bar displays current collection name, docs count, media count | Medium | -| BCK-13 | Grand progress bar shows overall collections progress | Green grand bar displays "Collection X of Y (Z%)" with cumulative docs/media | Medium | -| BCK-14 | Phase labels appear during processing | Text shows phase: "Reading documents…", "Packing collection…", etc. | Low | -| BCK-15 | Progress bar handles events without step/grand gracefully | If server sends minimal events, fallback pulse bar shows instead of broken UI | Low | - -### Import — Upload - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| BCK-16 | Upload with real progress tracking (live mode) | XHR upload fires progress events; progress bar fills during upload | Medium | -| BCK-17 | Invalid ZIP file shows error | Uploading a non-ZIP or corrupt file shows error message | High | - -### Import — Analysis & Restore - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| BCK-18 | Analysis table shows field warnings and non-restorable collections | Collections missing in target are greyed out; field diffs shown as warnings | High | -| BCK-19 | Merge toggle affects restore mode | Toggle shows correct description (merge on/off) | Medium | -| BCK-20 | Restore SSE flow completes and shows result | Restoring progress bar appears; done state shows restored docs/media count | High | - -### Auth & Permissions - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| BCK-21 | Non-admin user sees permission error | Visiting backup route shows "No administrator permission" message | High | -| BCK-22 | API responds 403 for tokens without `AllowBackup` | Server rejects requests from tokens missing the backup permission | High | - -### Edge Cases - -- Countdown timer expires → "Download link expired" message with retry button -- Abort during SSE (navigate away or reset) → clean state, no hanging request -- Expired restore token → server returns 404; frontend shows error -- Download token consumed → second attempt returns 410 (expired) -- Very large ZIP upload → progress bar shows accurate percentage - -## Accessibility - -- Progress bars have `role="progressbar"` with `aria-valuenow/min/max` -- Progress container uses `role="status"` and `aria-live="polite"` -- Tab buttons use `role="tab"` / `role="tablist"` with `aria-selected` -- Checkboxes in collection tables have visible labels and keyboard support - -## Test Automation - -- E2E tests: `tests/e2e/backup.spec.ts` (BCK-01 through BCK-09) -- Mock routes: `frontend/src/lib/mock/interceptor.ts` (lines 3022–3280) -- API mock data includes edge cases: expired tokens, schema warnings, non-existent collections diff --git a/src/content/docs/nova/test-plans/44-snapshots.md b/src/content/docs/nova/test-plans/44-snapshots.md deleted file mode 100644 index a650fd9..0000000 --- a/src/content/docs/nova/test-plans/44-snapshots.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: 'Snapshots (Admin)' ---- - -# Snapshots (Admin) - -## Scope - -- Admin UI: Snapshot overview (configs, entries, stats), snapshot trigger (SSE), download (SSE + ZIP blob), restore (SSE), delete, prune -- API communication via `snapshot-api.ts` (SSE parser, 10 endpoints) -- Mock interceptor coverage for all snapshot routes -- Access control: project admin permission required - -## Prerequisites - -- Run in mock mode (default) -- Use project t1 -- Admin token must have `AllowSnapshot: true` - -## Test Cases - -### Overview & Navigation - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-01 | Admin menu "Snapshots" button visible on project page | Button appears in admin dropdown; clicking navigates to `/projects/:id/snapshots` | High | -| SNP-02 | Direct deep-link to snapshots renders | Visiting `/projects/:id/snapshots` shows the snapshot view with configs tab active | High | -| SNP-03 | Three tabs exist (Configs, Entries, Stats) | Tabs are visible and switchable by click and by hash fragment | High | -| SNP-04 | Snapshots back button navigates to project | Clicking back arrow returns to project page | High | - -### Configs Tab - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-05 | Config list shows snapshot configurations | List renders with name, description, schedule, collections, media flag, retention info | High | -| SNP-06 | Trigger button starts snapshot SSE flow | Clicking trigger shows progress bar; SSE events arrive; final done state shows entry ID | High | -| SNP-07 | Trigger button is disabled for disabled configs | Config with `enabled: false` has disabled trigger button | Medium | - -### Entries Tab - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-08 | Entries table shows snapshot history | Table renders with rows per entry; columns show config name, status, date, collections, size | High | -| SNP-09 | Clicking a row expands detail view | Expanded detail shows per-collection breakdown (docs, media, size) | Medium | -| SNP-10 | Detail view data matches entry row summary | Aggregate totals in detail match the row-level display | Low | - -### Download — SSE + ZIP - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-11 | Download dialog opens with entry info | Clicking download icon on an entry row opens a dialog showing config name, date, size | High | -| SNP-12 | Download SSE flow shows progress | Clicking "Start download" shows spinner and phase messages (`packing` collections) | High | -| SNP-13 | Download SSE completes with download URL | After SSE finishes, dialog shows "Ready for download" with file size and countdown | High | -| SNP-14 | Download ZIP blob triggers browser download | Clicking "Download" button in done state triggers a file download with expected filename | High | -| SNP-15 | Expired download link shows expired state | Server returns 410; dialog shows "Download link expired" with retry button | Medium | - -### Restore - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-16 | Restore dialog opens with collection selection | Clicking restore icon opens a dialog with merge toggle and checkboxes for collections | High | -| SNP-17 | Restore SSE flow completes | Selecting collections and starting restore shows progress bar; done state shows restored doc/media count | High | -| SNP-18 | Merge toggle description changes | Toggling merge updates description text | Medium | - -### Delete, Stats & Prune - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-19 | Delete confirmation dialog appears | Clicking delete icon opens a ConfirmDialog with entry info | High | -| SNP-20 | Delete removes entry from list | Confirming delete removes row from table; API returns 200 | High | -| SNP-21 | Stats tab shows storage metrics | Stats view renders with entry count, total bytes, media bytes, config breakdown | Medium | -| SNP-22 | Prune button triggers retention cleanup | Clicking prune button sends API request; success message shown | Medium | - -### Auth & Permissions - -| ID | Description | Expected Result | Priority | -| --- | --- | --- | --- | -| SNP-23 | Non-admin user sees permission error | Visiting snapshots route shows "No administrator permission" message | High | - -### Edge Cases - -- SSE abort during download (navigate away or close dialog) → clean state, no hanging request -- SSE abort during trigger (navigate away) → clean state -- Expired download token → server returns 410; "Download link expired" message with retry -- Empty entries list → "No snapshots yet" placeholder shown -- Empty configs list → "No configurations" placeholder shown - -## Accessibility - -- Dialog has `role="dialog"` with `aria-label` -- Progress bars have `role="progressbar"` with `aria-valuenow/min/max` -- Tab list uses `role="tablist"` with `aria-selected` -- Buttons have descriptive titles or aria-labels diff --git a/src/content/docs/reference/index.md b/src/content/docs/reference/index.md deleted file mode 100644 index d66074e..0000000 --- a/src/content/docs/reference/index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: Referenz-Übersicht -description: Autogenerierte API- und Schema-Referenz. ---- - -> Diese Seiten werden generiert — **nicht von Hand editieren**. - -- **OpenAPI**: wird per `npm run sync:openapi` aus dem laufenden Tibi-Server (`/api/openapi.json`) hierher kopiert. -- **Hook-JS-API**: Typdefinitionen aus `tibi-server/tibi-types/` bzw. `tibi-types`. -- **Collection-Schema**: YAML-Schema-Referenz für Sammlungs-Konfiguration. - -Solange die Generatoren leer sind, steht hier nur diese Übersicht. diff --git a/src/content/docs/server/01-architecture.md b/src/content/docs/server/01-architecture.md deleted file mode 100644 index 6b2ff50..0000000 --- a/src/content/docs/server/01-architecture.md +++ /dev/null @@ -1,261 +0,0 @@ ---- -title: 'Architecture' ---- - -# Architecture - -## System Overview - -Tibi-Server is a Go-based headless CMS that provides dynamically configured REST APIs backed by MongoDB. Projects are defined via YAML configuration files that specify collections, hooks, jobs, and assets. Each project gets its own MongoDB database (prefixed with the server DB prefix). - -```text -┌─────────────────────────────────────────────────────────┐ -│ HTTP Clients │ -└─────────────┬───────────────────────────────┬───────────┘ - │ │ - ▼ ▼ -┌─────────────────────────┐ ┌──────────────────────────┐ -│ System API │ │ Collection API │ -│ /api/v1/login │ │ /api/v1/_/:project/... │ -│ /api/v1/user │ │ │ -│ /api/v1/project │ │ Dynamic CRUD per │ -│ /api/v1/llm/chat │ │ project config │ -│ /api/v1/audit │ │ │ -│ /api/v1/version │ │ Admin: │ -│ Admin: │ │ /reload │ -│ /shutdown │ │ /regenerate-search │ -│ /backup, /restore │ │ /backup, /restore │ -└─────────┬───────────────┘ └─────────┬────────────────┘ - │ │ - ▼ ▼ -┌─────────────────────────────────────────────────────────┐ -│ Gin Router / Middleware │ -│ Sentry → Gzip → Prometheus → RouterContext → CORS │ -└─────────────────────────┬───────────────────────────────┘ - │ - ┌───────────┴───────────┐ - ▼ ▼ -┌──────────────────────┐ ┌─────────────────────────────┐ -│ dbapi (System) │ │ Controller (Collections) │ -│ Generic CRUD for │ │ CollectionGet/Post/Put/ │ -│ User, Project │ │ Delete + BulkUpdate/ │ -│ │ │ BulkDelete with Hooks │ -└──────────┬───────────┘ └──────────┬──────────────────┘ - │ │ - │ ┌────────┴────────┐ - │ ▼ ▼ - │ ┌──────────────────┐ ┌────────────┐ - │ │ Hook Engine │ │ Validator │ - │ │ (goja JS) │ │ │ - │ │ 20+ packages │ └────────────┘ - │ └────────┬─────────┘ - │ │ - ▼ ▼ -┌─────────────────────────────────────────────────────────┐ -│ dbcrud Layer │ -│ Abstract DB Interface │ -└─────────────────────────┬───────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────┐ -│ MongoDB Driver │ -│ mongocrud implementation │ -└─────────────────────────────────────────────────────────┘ -``` - -## Directory Structure - -```text -tibi-server/ -├── cmd/server/main.go # Entry point: config loading, DB connection, HTTP server -├── internal/ -│ ├── config/server.go # ServerConfig struct, YAML + env variable loading -│ ├── controller/ # HTTP handlers -│ │ ├── setup.go # Router setup, route registration, middleware -│ │ ├── collection_get.go # GET handler with filter, projection, pagination -│ │ ├── collection_count.go # COUNT handler for /_count endpoint -│ │ ├── collection_post.go # POST handler with file upload -│ │ ├── collection_put.go # PUT handler with partial update -│ │ ├── collection_delete.go# DELETE handler with file cleanup -│ │ ├── collection_file.go # File serving with image filter support -│ │ ├── collection.go # bindCollection, handleFileInJSON (file upload logic) -│ │ ├── subst.go # $subst expression resolution for permission filters -│ │ ├── validator.go # Unified validateData (3 modes: Create/Update/Changeset) -│ │ ├── admin_reload.go # Hot-reload project config -│ │ ├── assets_get.go # Static asset serving -│ │ ├── audit_get.go # Audit log query endpoint -│ │ ├── llm_chat.go # LLM chat handler (streaming + non-streaming) -│ │ ├── metric.go # Prometheus middleware -│ │ ├── openapi_project_get.go # OpenAPI spec generation -│ │ ├── collection_bulk_update.go # Bulk update (PUT without ID, filter-based) -│ │ ├── collection_bulk_delete.go # Bulk delete (DELETE without ID, filter-based) -│ │ └── operators.go # Computed update operators ($inc, $mul, $unset) -│ ├── dbapi/ # Generic system-level CRUD framework -│ │ ├── handler.go # Generic GET/POST/PUT/DELETE for registered models -│ │ ├── filter.go # MongoDB filter validation and type conversion -│ │ ├── filter_valid_functions.go # Whitelist of allowed MongoDB operators -│ │ ├── login.go # JWT token creation and validation -│ │ └── register.go # Model registration (User, Project) -│ ├── dbcrud/ # Database abstraction layer -│ │ ├── interface.go # Database, Collection, Query interfaces -│ │ ├── model.go # Base Model (ID, InsertTime, UpdateTime, Ident) -│ │ ├── collection.go # Collection name resolution -│ │ └── validator.go # Struct-tag based validation -│ ├── hook/ # JavaScript hook context packages -│ │ ├── compatibility.go # Legacy function name mapping -│ │ ├── context_db.go # db.find, db.create, db.update, db.delete, db.updateMany, ... -│ │ ├── context_http.go # http.fetch, http.fetchStream -│ │ ├── context_fs.go # fs.readFile, fs.writeFile, fs.stat, ... -│ │ ├── context_smtp.go # smtp.sendMail -│ │ ├── context_response.go # response.write, response.writeSSE, response.flush -│ │ ├── context_channel.go # channel.subscribe, channel.send -│ │ └── ... # 20+ context packages total -│ ├── embedding/ # Embedding provider architecture (ONNX Runtime, tokenizer) -│ ├── image/ # Image processing (resize, effects, WebP) -│ ├── llm/ # LLM provider abstraction, quota management -│ ├── models/ # Domain models and business logic -│ │ ├── api.go # API struct: collections, hooks, jobs, CORS, hub -│ │ ├── project.go # Project model, config loading, setup -│ │ ├── collection.go # Collection model, CollectionConfig, fields, permissions -│ │ ├── permission.go # Permission checking logic -│ │ ├── hook.go # Hook execution (run, exception handling) -│ │ ├── eval_context.go # BuildEvalContext / PopulateJsData — central JS context builder -│ │ ├── field_permissions.go # Eval-based readonly/hidden field checks -│ │ ├── auth.go # AuthData extraction from JWT -│ │ ├── user.go # User model with MD5 password hashing -│ │ └── login.go # Login flow, refresh tokens -│ ├── rankfusion/ # Mode-independent Reciprocal Rank Fusion (N-way weighted RRF) -│ ├── realtime/ # SSE hub with channels, subscriptions, ring buffer -│ ├── suexec/ # Thread-based UID/GID isolation for multi-tenancy -│ ├── utils/ # Helpers: error formatting, search, file paths -│ ├── vectorindex/ # In-memory vector index (dense cosine + sparse inverted) -│ └── version/ # Build version info -├── tibi-types/ # TypeScript type definitions for hook context -├── examples/ # Example project configurations -├── vendor/ # Vendored Go dependencies -├── config.yml # Server configuration -├── Dockerfile # Production Docker image -├── Dockerfile.air # Development Docker image (hot-reload) -├── docker-compose.yml # Full stack: server + mongo + maildev + chisel -└── docker-compose-local.yml # MongoDB only for local development -``` - -## Request Lifecycle - -### Collection Request (e.g., GET /api/v1/_/myproject/articles) - -```text -1. HTTP Request - │ -2. Gin Middleware Chain - ├── Sentry (if configured): trace + error tracking - ├── Gzip compression (excludes images, videos, PDFs, /metrics) - ├── Prometheus: start timing - ├── RouterContext: inject global project API context - └── CORS: resolve config (collection → project → server level) - │ -3. Route Handler - ├── Create new DB session - ├── Resolve project namespace from URL path - ├── Resolve collection config from URL path - │ -4. Permission Check (MethodIsAllowed) - ├── Check token-based permissions (header/query) - ├── Check public permissions - ├── Check admin role (role=0 → always allowed) - ├── Check user permissions (role=1 + project assignment) - └── Check custom permissions (namespace:permName) - │ -5. Pre-Filter (getCollectionPreFilter) - ├── Apply permission filter (e.g., {published: true}) - └── Resolve $subst expressions (auth.id, auth.role, etc.) - │ -6. Hook Pipeline (per method) - │ GET: request → read → [db query] → return - │ COUNT: read → [db count] (no return hook) - │ POST: bind → validate → create → [db insert] → return - │ PUT: bind → validate → update → [db update] → return - │ DELETE: delete → [db delete] → return - │ - ├── Each hook step: - │ ├── Load compiled JS program - │ ├── Build hook context (api, project, collection, request, ...) - │ ├── Inject context packages (db, http, fs, smtp, ...) - │ ├── Execute JS with timeout - │ └── Process return value (data override, results override) - │ - ├── Hook can throw HookException: - │ ├── {status, json} → JSON response - │ ├── {status, html} → HTML response - │ ├── {status, string} → text response - │ ├── {status, bytes} → binary response - │ └── {status, file} → file download - │ -7. Database Operation - ├── ReadCollection / CreateDocument / UpdateDocument / DeleteDocument - └── File system operations (upload, delete) - │ -8. Audit Log (if enabled) - ├── Record action, user, document snapshot, changes - └── Write async to audit collection with TTL index - │ -9. Response - ├── NaN/Inf values converted to null - ├── JSON serialization - └── Prometheus: record duration -``` - -## Data Model - -### Base Model (dbcrud.Model) - -Every document in the system inherits from the base model: - -```go -type Model struct { - ID interface{} // MongoDB ObjectID - InsertTime *time.Time // Set automatically on create - UpdateTime *time.Time // Set automatically on update - Ident []string // Display identifier (e.g., username, document title) -} -``` - -### System Models - -| Model | Database | Collection | Description | -| -------------- | ---------- | ------------- | ---------------------------------------------- | -| `User` | `{prefix}` | `user` | Admin/user accounts with roles and permissions | -| `Project` | `{prefix}` | `project` | Project definitions with config file paths | -| `AuditLog` | `{prefix}` | `auditlog` | System-wide audit trail | -| `LLMUsageLog` | `{prefix}` | `llmusagelog` | LLM token usage tracking | - -### Project Models - -| Model | Database | Collection | Description | -| -------------- | ---------------------- | ------------------- | ------------------------------ | -| `Collection` | `{prefix}_{namespace}` | `{collectionName}` | Dynamic, schema-less documents | - -Each project gets its own database named `{prefix}_{namespace}`, where `prefix` is the server DB prefix (default: `tibi`) and `namespace` is defined in the project's YAML config. - -## Middleware Chain - -| Order | Middleware | Source | Purpose | -| ----- | ------------- | --------------------------- | ------------------------------------- | -| 1 | Sentry | `sentrygin.New()` | Error tracking, transaction tracing | -| 2 | Sentry Scope | Custom | Set user IP, start spans | -| 3 | Gzip | `gzip.Gzip()` | Compression (excludes binary formats) | -| 4 | Prometheus | `measureResponseDuration()` | Request timing metrics | -| 5 | RouterContext | Custom | Inject project API context | -| 6 | CORS | `cors.New()` | Cross-origin configuration | - -## Key Design Decisions - -1. **Schema-less collections**: Collection documents use `map[string]interface{}` (BSON inline) rather than fixed Go structs. Field definitions in YAML config are used for validation and type checking but don't restrict storage. - -2. **Per-project databases**: Each project gets its own MongoDB database, providing data isolation between projects. - -3. **JavaScript hooks via goja**: Hooks run in a Go-embedded JavaScript engine (goja), not Node.js. This means ECMAScript 5.1 compatibility with some ES6+ features. No npm modules available — all functionality comes from built-in context packages. - -4. **Suexec isolation**: On Linux, hook execution and file operations run under the UID/GID of the project's config file owner, providing OS-level multi-tenant isolation. - -5. **Session-per-request**: Each HTTP request creates a new MongoDB session that is closed after the request completes. diff --git a/src/content/docs/server/02-configuration.md b/src/content/docs/server/02-configuration.md deleted file mode 100644 index 8f40b9b..0000000 --- a/src/content/docs/server/02-configuration.md +++ /dev/null @@ -1,622 +0,0 @@ ---- -title: 'Configuration' ---- - -# Configuration - -Tibi-Server uses a layered configuration system: a YAML config file with environment variable overrides. - -## Server Configuration (`config.yml`) - -The server configuration file is loaded from the working directory at startup. - -### Complete Reference - -```yaml -db: - dial: "mongodb://localhost:27017" # MongoDB connection string - prefix: "tibi" # Database name prefix - logging: "" # Enable DB query logging (any non-empty value) - -api: - port: 8080 # HTTP server port - maxBodySize: "100MB" # Max request body size (default: 100MB). Supports B, KB, MB, GB, TB. - # Cascade: Server (hard cap) → Project (default) → Collection (override) - # Project-level: set maxBodySize in project config.yml (can only reduce) - # Collection-level: set maxBodySize in collection config.yml (can override project, but not server) - jwtSecret: "your-jwt-secret" # HMAC-SHA256 secret for JWT signing - secureCookies: true # Set Secure flag on cookies (default: true, set false for local HTTP dev) - accessTokenLifetime: "60m" # JWT access token lifetime (Go duration, default: 60m) - refreshTokenLifetime: "10h" # Refresh token / cookie lifetime (Go duration, default: 10h) - adminTokens: # Server-level admin tokens with granular permissions - - token: "agent-token-xyz" # Token string (sent via X-Admin-Token header or Authorization: Bearer) - label: "ci-deploy" # Optional label for audit logging (identifies the token purpose) - permissions: # List of granted permissions - - project # Shorthand for project.* (all CRUD) - - project.reload # Reload project config - - user # Shorthand for user.* (all CRUD) - - server.shutdown # Server shutdown - - namespace.myproject # All collections in myproject (all actions) - - namespace.*.articles.read # Read articles in any project (wildcard) - - token: "readonly-token" - label: "monitoring" # Shows up in audit logs as tokenLabel - permissions: - - project.read # Only read projects - - user.read # Only read users - - namespace.*.*.read # Readonly on all collections in all projects - -mail: - host: "smtp.example.com:587" # SMTP server host:port - username: "user@example.com" # SMTP username - password: "password" # SMTP password - -security: - allowAbsolutePaths: false # Allow absolute file paths in hooks/config - allowUpperPaths: false # Allow path traversal above config directory - -sentry: - dsn: "" # Sentry DSN for error tracking - environment: "production" # Sentry environment tag - serverName: "tibi-1" # Sentry server name - enableTracing: true # Enable performance tracing - debug: false # Sentry debug mode - -cors: - allowOrigins: # Allowed origins (server-level default) - - "https://example.com" - allowCredentials: false # Allow credentials (cookies, auth headers) - allowHeaders: # Allowed request headers - - "Content-Type" - - "X-Auth-Token" - allowMethods: # Allowed HTTP methods - - "GET" - - "POST" - - "PUT" - - "DELETE" - exposeHeaders: [] # Headers exposed to the browser - maxAge: 0 # Preflight cache duration (seconds) - -response: - error: - message: "" # Fixed error message for clients (hides details) - stack: false # Include stack traces in error responses - -llm: - providers: - - name: "openai" # Provider identifier - type: "openai" # "openai" or "openai-compatible" - baseURL: "https://api.openai.com/v1" # API base URL (auto-set for "openai" type) - apiKey: "sk-..." # API key (prefer env variable) - defaultModel: "gpt-4" # Default model - models: # Available models - - "gpt-4" - - "gpt-3.5-turbo" - maxTokensPerRequest: 4096 # Max tokens per request (optional) - -audit: - enabled: false # Enable audit logging - defaultTTL: "720h" # Default audit log retention (30 days) - defaultLimit: 50 # Default query limit for audit GET (fallback: 50) - maxLimit: 10000 # Maximum query limit for audit GET (fallback: 10000) - -ratelimit: - enabled: true # Enable login rate limiting (default: true) - loginInitialDelay: "1s" # Initial delay after first failed login attempt - loginMaxDelay: "60s" # Maximum delay cap (exponential backoff) - loginResetAfter: "15m" # Reset attempt counter after inactivity period - -embedding: - providers: - - name: "bge-m3" # Provider identifier (referenced in collection search config) - type: "native" # "native" (ONNX), "openai", "openai-compatible", "ollama" - modelPath: "/models/bge-m3" # Path to ONNX model directory (type: native) - dimensions: 1024 # Embedding dimensions (required for native) - - name: "e5-small" - type: "native" - modelPath: "/models/multilingual-e5-small" - dimensions: 384 - - name: "openai-embed" # Example: external OpenAI provider - type: "openai" - model: "text-embedding-3-small" # Model name for API calls - apiKey: "sk-..." # API key (prefer env variable) - baseURL: "https://api.openai.com/v1" # API base URL - dimensions: 1536 - -hook: - cache: - maxEntries: 10000 # Max cache entries per project before LRU eviction (default: 10000) - defaultTTL: "0s" # Default TTL for cache.set() when ttl=0 (default: no expiry) - ratelimit: - backoffInitialDelay: "1s" # Backoff initial delay after first failure (default: 1s) - backoffMaxDelay: "60s" # Backoff maximum delay cap (default: 60s) - backoffResetAfter: "15m" # Backoff reset after inactivity (default: 15m) - windowSize: "60s" # Sliding window duration (default: 60s) - windowMaxHits: 60 # Max hits per sliding window (default: 60) -``` - -### Default Values (no config file) - -When no config file is found and `defaultFallback` is true: - -| Setting | Default | -| ------------------------ | --------- | -| `api.port` | `8080` | -| `db.prefix` | `"tibi"` | -| `api.maxBodySize` | `"100MB"` | -| `response.error.stack` | `true` | - -## Environment Variable Overrides - -Environment variables take precedence over config file values. - -| Environment Variable | Config Path | Type | Notes | -| ----------------------------- | ----------------------------- | ------ | ----------------------------------- | -| `DB_DIAL` | `db.dial` | string | MongoDB connection string | -| `MGO_DIAL` | `db.dial` | string | Legacy alias for `DB_DIAL` | -| `DB_PREFIX` | `db.prefix` | string | Database name prefix | -| `DB_LOGGING` | `db.logging` | string | Any non-empty value enables logging | -| `API_UPLOAD_MAX_BODY_SIZE` | `api.upload.maxBodySize` | string | e.g. `"100MB"` (B, KB, MB, GB, TB) | -| `API_PORT` | `api.port` | int | HTTP port (panics if ≤ 0) | -| `PORT` | `api.port` | int | Fallback for `API_PORT` | -| `API_JWTSECRET` | `api.jwtSecret` | string | JWT signing secret | -| `API_SECURE_COOKIES` | `api.secureCookies` | bool | Secure flag on cookies (default: true) | -| `API_ACCESS_TOKEN_LIFETIME` | `api.accessTokenLifetime` | string | Go duration, e.g. `"60m"` (default) | -| `API_REFRESH_TOKEN_LIFETIME` | `api.refreshTokenLifetime` | string | Go duration, e.g. `"10h"` (default) | -| `MAIL_HOST` | `mail.host` | string | SMTP host:port | -| `SECURITY_ALLOWABSOLUTEPATHS` | `security.allowAbsolutePaths` | bool | `"true"` or `"false"` | -| `SECURITY_ALLOWUPPERPATHS` | `security.allowUpperPaths` | bool | `"true"` or `"false"` | -| `RESPONSE_ERROR_STACK` | `response.error.stack` | bool | Show stack traces | -| `RESPONSE_ERROR_MESSAGE` | `response.error.message` | string | Fixed error message | -| `AUDIT_ENABLED` | `audit.enabled` | bool | Enable audit logging | -| `AUDIT_DEFAULT_TTL` | `audit.defaultTTL` | string | Go duration (e.g., `"720h"`) | -| `AUDIT_DEFAULT_LIMIT` | `audit.defaultLimit` | int | Audit GET default page size | -| `AUDIT_MAX_LIMIT` | `audit.maxLimit` | int | Audit GET max page size | -| `RATELIMIT_ENABLED` | `ratelimit.enabled` | bool | Enable login rate limiting | -| `RATELIMIT_LOGIN_INITIAL_DELAY` | `ratelimit.loginInitialDelay` | string | Go duration (e.g., `"1s"`) | -| `RATELIMIT_LOGIN_MAX_DELAY` | `ratelimit.loginMaxDelay` | string | Go duration (e.g., `"60s"`) | -| `RATELIMIT_LOGIN_RESET_AFTER` | `ratelimit.loginResetAfter` | string | Go duration (e.g., `"15m"`) | -| `HOOK_CACHE_MAX_ENTRIES` | `hook.cache.maxEntries` | int | Max cache entries per project | -| `HOOK_CACHE_DEFAULT_TTL` | `hook.cache.defaultTTL` | string | Go duration (e.g., `"5m"`) | -| `HOOK_RATELIMIT_BACKOFF_INITIAL_DELAY` | `hook.ratelimit.backoffInitialDelay` | string | Go duration (e.g., `"1s"`) | -| `HOOK_RATELIMIT_BACKOFF_MAX_DELAY` | `hook.ratelimit.backoffMaxDelay` | string | Go duration (e.g., `"60s"`) | -| `HOOK_RATELIMIT_BACKOFF_RESET_AFTER` | `hook.ratelimit.backoffResetAfter` | string | Go duration (e.g., `"15m"`) | -| `HOOK_RATELIMIT_WINDOW_SIZE` | `hook.ratelimit.windowSize` | string | Go duration (e.g., `"60s"`) | -| `HOOK_RATELIMIT_WINDOW_MAX_HITS` | `hook.ratelimit.windowMaxHits` | int | Max hits per sliding window | -| `LLM__APIKEY` | `llm.providers[name].apiKey` | string | Provider API key (uppercase name) | -| `EMBEDDING__APIKEY` | `embedding.providers[name].apiKey` | string | Embedding provider API key (uppercase name) | -| `EMBEDDING__MODELPATH` | `embedding.providers[name].modelPath` | string | ONNX model directory path (uppercase name) | -| `EMBEDDING__DOCUMENTPREFIX` | `embedding.providers[name].documentPrefix` | string | Global prefix applied to document embeddings (uppercase name) | -| `EMBEDDING__QUERYPREFIX` | `embedding.providers[name].queryPrefix` | string | Global prefix applied to query embeddings (uppercase name) | - -### LLM API Key Convention - -LLM providers can be configured completely via environment variables (without changing `config.yml`) using `LLM_PROVIDERS`: - -```bash -# Register two providers -export LLM_PROVIDERS=openai,local-llama - -# Configure the "openai" provider -export LLM_OPENAI_APIKEY=sk-... - -# Configure the "local-llama" provider -export LLM_LOCAL-LLAMA_TYPE=openai-compatible -export LLM_LOCAL-LLAMA_BASEURL=http://localhost:11434/v1 -export LLM_LOCAL-LLAMA_MODELS=llama3,mistral -export LLM_LOCAL-LLAMA_DEFAULTMODEL=llama3 -``` - -All available config fields (`APIKEY`, `TYPE`, `BASEURL`, `DEFAULTMODEL`, `MODELS`) can be overridden via `LLM__`. - -### Embedding Provider Environment Convention - -Embedding providers are configured in exactly the same way as LLM providers. Rather than `llm:`, you provide an `embedding:` block in your server's `config.yml`. - -Example in `config.yml`: -```yaml -embedding: - providers: - - name: bge-m3 - type: native - model: bge-m3 - dimensions: 1024 - # modelPath: /assets/models/bge-m3 - # documentPrefix: "" - # queryPrefix: "" - - name: text-embedding-ada-002 - type: openai - model: text-embedding-ada-002 - dimensions: 1536 - # apiKey: sk-... -``` - -Just like LLMs, you can provision Embedding providers **purely via environment variables** (very useful for Docker setups): - -```bash -# Register two providers via comma-separated list -export EMBEDDING_PROVIDERS=openai-embed,bge-m3 - -# Configure "openai-embed" -export EMBEDDING_OPENAI-EMBED_TYPE=openai -export EMBEDDING_OPENAI-EMBED_MODEL=text-embedding-ada-002 -export EMBEDDING_OPENAI-EMBED_DIMENSIONS=1536 -export EMBEDDING_OPENAI-EMBED_APIKEY=sk-... - -# Configure "bge-m3" -export EMBEDDING_BGE-M3_TYPE=native -export EMBEDDING_BGE-M3_MODELPATH=/models/bge-m3 -export EMBEDDING_BGE-M3_DIMENSIONS=1024 -export EMBEDDING_BGE-M3_DOCUMENTPREFIX="search_document: " -export EMBEDDING_BGE-M3_QUERYPREFIX="search_query: " -``` - -Available config fields: `TYPE`, `MODEL`, `MODELPATH`, `DIMENSIONS`, `BASEURL`, `APIKEY`, `DOCUMENTPREFIX`, `QUERYPREFIX`. -**Note:** The collections within your projects will reference these providers by their `name`. - -Default type for embedding providers (if omitted in config) is `"native"` (ONNX). - ---- - -## Project Configuration - -Each project has a YAML config file referenced in the project's `configFile` field in the database. The config file defines all collections, hooks, jobs, and assets for that project. - -### Complete Project Reference - -```yaml -# Project namespace (used as database suffix and URL path segment) -# Must match: [a-zA-Z0-9_]+ -namespace: myproject - -# Arbitrary metadata accessible in hooks via context.project().meta -meta: - title: "My Project" - version: "1.0" - -# Collection definitions -collections: - - name: articles # Collection name (URL path segment, MongoDB collection name) - defaultLanguage: "german" # Default language for text indexes - uploadPath: "./media" # File upload directory (relative to config dir) - - fields: # Field definitions (see Collections docs) - - name: title - type: string - validator: - required: true - - name: image - type: file - validator: - maxFileSize: "5MB" # Max file size for this field (supports B, KB, MB, GB, TB) - - permissions: # Permission sets (see Authentication docs) - public: - methods: - get: true - filter: - published: true - user: - methods: - get: true - post: true - put: true - delete: true - - projections: # Named projections (field subsets) - default: - select: - title: 1 - image: 1 - detailed: - select: - title: 1 - image: 1 - content: 1 - - imageFilter: # Named image filter presets - thumbnail: - - width: 200 - height: 200 - fit: true - quality: 80 - hero: - - width: 1200 - height: 600 - fill: true - anchor: center - - hooks: # Lifecycle hooks (see Hooks docs) - get: - request: - type: javascript - file: "hooks/get_request.js" - timeout: 5000 - post: - create: - type: javascript - file: "hooks/post_create.js" - - indexes: # Custom MongoDB indexes - - name: title_text - key: - - "$text:title" - defaultLanguage: german - - name: date_sort - key: - - "-publishDate" - background: true - - strictFields: true # Reject unknown fields not in schema (default: inherit from project) - - cors: # Collection-level CORS override - merge: true - allowOrigins: - - "https://special.example.com" - - audit: # Collection-level audit config - enabled: true - actions: - - create - - update - - delete - - queryLimits: # Query result limits for GET list endpoint - defaultLimit: 100 # Default limit when no ?limit= param (fallback: 10000) - maxLimit: 500 # Maximum allowed limit (fallback: 10000) - -audit: # Project-level audit (opt-in, default false) - enabled: true # required in addition to server audit.enabled - -# Cron jobs -jobs: - - cron: "0 */5 * * *" # Every 5 minutes - type: javascript - file: "jobs/cleanup.js" - timeout: 30000 # Milliseconds (30 s) - meta: - description: "Cleanup old drafts" - -# Static asset directories -assets: - - name: static - path: "./public" - - name: uploads - path: "./uploads" - -# Reject unknown fields for all collections by default (default: false) -strictFields: true - -# Project-level CORS configuration -cors: - merge: true # Merge with server CORS (true) or replace (false) - allowOrigins: - - "https://myapp.example.com" - allowCredentials: true - allowHeaders: - - "X-Custom-Header" - -# Admin configuration -admin: - tokens: - - token: "deploy-token-abc123" - allowReload: true - -# Hook settings (project-level overrides) -hook: - cache: - maxEntries: 5000 # Override server default for this project - defaultTTL: "5m" # Override default TTL for this project - ratelimit: - backoffInitialDelay: "2s" # Override backoff initial delay - backoffMaxDelay: "120s" # Override backoff max delay - backoffResetAfter: "30m" # Override backoff reset period - windowSize: "30s" # Override sliding window duration - windowMaxHits: 30 # Override max hits per window - -# Legacy compatibility settings -legacy: - httpCodes: true # true (default): use original status codes for backward compatibility - # false: use RFC-compliant HTTP status codes (201, 204, 401, 409, 422) -``` - -### Environment Variable Substitution in Project Config - -Project config files support a companion `.env` file. If `config.yml.env` exists alongside `config.yml`, its variables are substituted into the YAML before parsing: - -```bash -# config.yml.env -API_KEY=secret-key-123 -WEBHOOK_URL=https://example.com/webhook -``` - -```yaml -# config.yml - use ${VAR} syntax -meta: - apiKey: "${API_KEY}" - webhookUrl: "${WEBHOOK_URL}" -``` - -### YAML Include Support - -Project configs support including other YAML files for modular collection definitions: - -```yaml -# config.yml -namespace: myproject -collections: !include collections/*.yml -``` - -This allows splitting collection configs into separate files under a `collections/` directory. - ---- - -## Legacy HTTP Status Codes - -By default, Tibi-Server uses legacy HTTP status codes for backward compatibility. To opt into RFC-compliant status codes, set `legacy.httpCodes: false` in the project config: - -```yaml -legacy: - httpCodes: false -``` - -### Status Code Mapping - -| Scenario | Legacy (`true`, default) | Modern (`false`) | -|---|---|---| -| POST/create success | 200 OK | **201** Created | -| Single DELETE success | 200 OK + body | **204** No Content | -| Authentication missing | 403 Forbidden | **401** Unauthorized | -| Validation error | 400 Bad Request | **422** Unprocessable Entity | -| Duplicate key conflict | 500 Internal Server Error | **409** Conflict | -| Field permission denied (readonly/hidden) | 400 Bad Request | **403** Forbidden | - -**Notes:** -- Bulk DELETE always returns 200 with `{"deleted": N}` regardless of the setting. -- Bulk POST returns 201 in modern mode (with `{"created": N, "ids": [...]}`). -- The default is `true` — existing projects are unaffected unless explicitly opted in. -- The setting is per-project, allowing gradual migration. - ---- - -## CORS Configuration Hierarchy - -CORS is configured at three levels, with each level optionally merging with or replacing the parent: - -```text -Server CORS (config.yml → cors) - │ - ├── merge: false → Replace entirely - │ - ▼ -Project CORS (project config → cors) - │ - ├── merge: true → Append to server origins/headers/methods - ├── merge: false → Replace entirely - │ - ▼ -Collection CORS (collection config → cors) - │ - └── merge: true → Append to project+server origins/headers/methods -``` - -### Resolution Order (per request) - -1. Check if there's a collection-specific CORS config → `CORSMap["/collectionName"]` -2. Fall back to project-level CORS → `CORSMap["/"]` -3. Fall back to server-level CORS → `CORSMap[""]` - -### Merge Behavior - -When `merge: true`: - -- `allowOrigins` are **appended** to the parent config -- `allowHeaders` are **appended** to the parent config -- `allowMethods` are **appended** to the parent config -- `exposeHeaders` are **replaced** (not appended — see [ISSUES.md](ISSUES.md)) -- `allowCredentials` is **replaced** -- `maxAge` is **set to 0** (always) -- Duplicates are automatically removed - -### CORSConfig Struct - -```go -type CORSConfig struct { - Merge bool `yaml:"merge"` - AllowOrigins []string `yaml:"allowOrigins"` - AllowCredentials bool `yaml:"allowCredentials"` - AllowHeaders []string `yaml:"allowHeaders"` - AllowMethods []string `yaml:"allowMethods"` - ExposeHeaders []string `yaml:"exposeHeaders"` - MaxAge int `yaml:"maxAge"` -} -``` - ---- - -## Collection Configuration Reference - -### CollectionConfig Struct - -```go -type CollectionConfig struct { - Name string // Collection/MongoDB collection name - DefaultLanguage *string // Default language for text indexes - UploadPath *string // File upload directory (relative or absolute) - Permissions map[string]PermissionSet // Permission sets (public, user, token:*, custom) - Projections map[string]ProjectionConfig // Named field projections - Meta map[string]interface{} // Arbitrary metadata - Fields []CollectionField // Field definitions - ImageFilter map[string][]image.FilterParams // Named image filter presets - Hooks map[string]map[string]HookConfig // Lifecycle hooks [method][step] - Indexes []CollectionIndex // Custom MongoDB indexes - CORS *CORSConfig // Collection-level CORS - Audit *AuditCollectionConfig // Collection-level audit config - Bulk *BulkConfig // Bulk operation config - QueryLimits *QueryLimitsConfig // Query result limits (defaultLimit, maxLimit) -} -``` - -### CollectionIndex - -```yaml -indexes: - - name: "unique_email" # Index name - key: # Index keys (prefix with - for descending, $text: for text) - - "email" - unique: true # Unique constraint - sparse: true # Skip documents without the indexed field - background: true # Build in background - dropDups: false # Drop duplicate entries (deprecated in MongoDB 3.0+) - defaultLanguage: "english" # Language for text indexes - languageOverride: "lang" # Field that overrides the language -``` - -### Field-Level Indexes - -Fields can define indexes directly via the `index` property: - -```yaml -fields: - - name: email - type: string - index: - - single # Regular index on this field - - unique # Unique index - - text # Text search index - - sparse # Sparse index -``` - -### Index Auto-Management - -On project setup/reload: - -1. All configured indexes are created via `EnsureIndex` -2. Indexes that exist in MongoDB but are no longer in the config are automatically dropped (`fixIndexes`) -3. The `_id_` index is never dropped - ---- - -## Admin Configuration - -### Project-Level Admin Tokens - -Each project can define admin tokens for project-specific operations: - -```yaml -admin: - tokens: - - token: "deploy-token-abc123" - allowReload: true - allowRegenerateSearch: true -``` - -Available permissions: -- `allowReload` – allows reloading the project configuration -- `allowRegenerateSearch` – allows regenerating search indexes (`_search` data) - -Usage: - -```bash -curl -X POST http://localhost:8080/api/v1/_/myproject/_/admin/reload \ - -H "Authorization: Bearer deploy-token-abc123" -``` diff --git a/src/content/docs/server/03-api-endpoints.md b/src/content/docs/server/03-api-endpoints.md deleted file mode 100644 index 99f5612..0000000 --- a/src/content/docs/server/03-api-endpoints.md +++ /dev/null @@ -1,2410 +0,0 @@ ---- -title: 'API Endpoints' ---- - -# API Endpoints - -All API endpoints are prefixed with `/api/v1`. Authentication is via the `X-Auth-Token` header containing a JWT token (obtained from the login endpoint). - -## System Endpoints - -### Version - -```text -GET /api/v1/version -``` - -Returns the server version. - -**Response:** - -```json -{ - "version": "1.0.0" -} -``` - -**Auth:** None required. - ---- - -### Login - -```text -POST /api/v1/login -``` - -Authenticates a user and returns a JWT token. - -**Request Body:** - -```json -{ - "username": "admin", - "password": "admin" -} -``` - -**Response (200):** - -```json -{ - "token": "eyJhbGciOiJIUzI1NiIs...", - "user": { - "id": "507f1f77bcf86cd799439011", - "username": "admin", - "role": 0, - "permissions": [] - } -} -``` - -**JWT Token Claims:** - -```json -{ - "object": { "id": "...", "username": "...", "role": 0, "permissions": [] }, - "type": "User", - "exp": 1700000000 -} -``` - -- Token expiration: **60 minutes** -- Signing method: HMAC-SHA256 - -**Refresh Token:** - -On successful login, a refresh token cookie is set: - -- Name: `tibi-refresh-token` -- HttpOnly: `true` -- SameSite: `Strict` -- Expires: **10 hours** - -To refresh a token, send an empty POST to `/api/v1/login` (with the cookie present, no body required). - -**Logout:** - -```text -POST /api/v1/login?logout=1 -``` - -Clears the refresh token cookie. - -**Error Responses:** - -| Status | Condition | -| ------ | ------------------------------- | -| 403 | Invalid credentials | -| 403 | Invalid/expired refresh token | - ---- - -### Metadata - -```text -GET /api/v1/_meta -``` - -Returns metadata about all registered system models (User, Project), including field definitions and indexes. - -**Auth:** None required. - -**Response (200):** - -```json -{ - "models": { - "user": [ - { - "name": "Username", - "json": "username,omitempty", - "type": "string", - "kind": "ptr", - "validator": ["required", "alphanum"] - } - ], - "project": [ - { - "name": "Name", - "json": "name", - "type": "string", - "kind": "ptr", - "validator": ["required"] - } - ] - } -} -``` - -Each field entry contains: - -| Field | Description | -| ----------- | ------------------------------------------------------------------ | -| `name` | Go struct field name | -| `json` | JSON tag (serialization name + options) | -| `type` | Go type name | -| `kind` | Go reflect kind (`ptr`, `string`, `int`, `slice`, `struct`, etc.) | -| `validator` | Validation rules (if any) | -| `structOf` | Nested fields (for struct types) | -| `sliceOf` | Element type or nested fields (for slice types) | - -#### Get Single Model Metadata - -```text -GET /api/v1/_meta/:modelName -``` - -Returns metadata for a single model (e.g. `user` or `project`). - -**Response (200):** Array of field metadata objects (same format as above). - ---- - -### Metrics - -```text -GET /metrics -``` - -> **Note:** This endpoint is outside `/api/v1`. - -Returns Prometheus metrics. See [OpenAPI & Metrics](13-openapi-metrics.md). - -**Auth:** None required. - ---- - -## Org Management - -All org endpoints require JWT authentication. Only SystemAdmins can create, update, or delete orgs. Non-admins see only their own orgs. - -For detailed documentation on roles, visibility rules, and cascade behavior, see [Orgs & Teams](18-orgs-teams.md). - -GET endpoints support `?lookup=` to resolve foreign-key references inline (see [System Endpoint Lookup](#system-endpoint-lookup)). - -```text -GET /api/v1/org # List (filtered by membership) -GET /api/v1/org/:id # Get single -POST /api/v1/org # Create (admin only) -PUT /api/v1/org/:id # Update (admin only) -DELETE /api/v1/org/:id # Delete (admin only, cascades) -``` - ---- - -## Team Management - -Team endpoints require JWT authentication. SystemAdmins can manage all teams. OrgAdmins can manage teams within their orgs. - -GET endpoints support `?lookup=` to resolve foreign-key references inline (see [System Endpoint Lookup](#system-endpoint-lookup)). - -```text -GET /api/v1/team # List (filtered by membership/org) -GET /api/v1/team/:id # Get single -POST /api/v1/team # Create (admin or orgAdmin) -PUT /api/v1/team/:id # Update (admin or orgAdmin) -DELETE /api/v1/team/:id # Delete (admin or orgAdmin, cascades) -``` - ---- - -## User Management - -All user endpoints require JWT authentication. Admin role (0) can manage all users. OrgAdmins can manage users within their orgs. User role (1) can only GET and PUT their own profile (limited fields). - -For org/team-scoped user management, see [Orgs & Teams](18-orgs-teams.md#user-management-orgteam-scoped). - -### List Users - -```text -GET /api/v1/user -``` - -**Query Parameters:** - -| Parameter | Type | Description | -| ---------- | ------------- | ----------------------------------------------------- | -| `filter` | JSON string | MongoDB filter query | -| `limit` | int | Max results to return | -| `offset` | int | Skip N results | -| `sort` | string | Sort field(s), prefix `-` for descending | -| `sort[]` | string[] | Multiple sort fields | -| `count` | `1` or `true` | Include count in `X-Results-Count` header | -| `_search` | string | Full-text search across username, meta fields | -| `lookup` | string / JSON | Resolve foreign-key references inline (see [System Endpoint Lookup](#system-endpoint-lookup)) | -| `lookup[]` | string[] | Alternative: multiple lookup parameters (bracket notation) | - -**Example:** - -```bash -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/user?limit=10&sort=-insertTime" -``` - -### Get User - -```text -GET /api/v1/user/:id -``` - -**Response (200):** - -```json -{ - "id": "507f1f77bcf86cd799439011", - "insertTime": "2024-01-15T10:30:00Z", - "updateTime": "2024-01-16T14:22:00Z", - "username": "admin", - "role": 0, - "permissions": ["myproject:editor"], - "orgs": [ - { "orgId": "507f1f77bcf86cd799439013", "role": "admin" } - ], - "teams": [ - { "teamId": "507f1f77bcf86cd799439014", "role": "member" } - ], - "primaryOrgId": "507f1f77bcf86cd799439013", - "meta": { - "email": "admin@example.com", - "prename": "John", - "surname": "Doe" - }, - "llmBudgets": [ - { - "provider": "openai", - "tokenBudget": 100000, - "tokensUsed": 5000, - "periodStart": "2024-01-01T00:00:00Z", - "periodEnd": "2024-02-01T00:00:00Z" - } - ] -} -``` - -> **Note:** The `password` field is **never** included in responses (stripped by custom JSON marshaller). - -**Response (404):** - -```json -{ - "error": "not found", - "errorCode": "20240115103000-a1b2" -} -``` - -### Create User - -```text -POST /api/v1/user -``` - -**Request Body:** - -```json -{ - "username": "newuser", - "password": "securepass", - "role": 1, - "permissions": ["myproject:editor"], - "meta": { - "email": "user@example.com", - "prename": "John", - "surname": "Doe" - } -} -``` - -**Response (200):** Returns the created user (same format as Get User). - -**Auth:** Admin only. - -### Update User - -```text -PUT /api/v1/user/:id -``` - -Partial update — only send fields to change. Username cannot be changed. Only admins can change `role`, `permissions`, and `llmBudgets`. OrgAdmins can modify `orgs`, `teams`, and `primaryOrgId` within their scope (see [Orgs & Teams — Field Protection](18-orgs-teams.md#field-protection)). - -**Response (200):** Returns the updated user (same format as Get User). - -**Auth:** Admin, or user editing own profile (limited fields). - -### Delete User - -```text -DELETE /api/v1/user/:id -``` - -**Response (200):** - -```json -{ - "message": "ok" -} -``` - -**Auth:** Admin only. - ---- - -## Project Management - -All project endpoints require JWT authentication with admin role. Non-admins see only projects belonging to their orgs. - -GET endpoints support `?lookup=` to resolve foreign-key references inline (see [System Endpoint Lookup](#system-endpoint-lookup)). - -### List Projects - -```text -GET /api/v1/project -``` - -Returns projects with enriched data: - -- `api`: project API info (online status, errors, collections with field definitions and indexes, jobs) -- `yourPermissions`: computed permissions per collection for the current user - -User role (1) only sees projects belonging to orgs they are a member of (filtered by `orgId`). Users with no org membership see no projects. - -**Query Parameters:** Same as User endpoints (`filter`, `limit`, `offset`, `sort`, `count`, `_search`). - -**Auth:** JWT (Admin sees all, User sees org-scoped projects) or Admin-Token with `project.read` permission. - -### Get Project - -```text -GET /api/v1/project/:id -``` - -**Response (200):** - -```json -{ - "id": "507f1f77bcf86cd799439011", - "insertTime": "2024-01-15T10:30:00Z", - "updateTime": "2024-01-16T14:22:00Z", - "name": "My Blog", - "description": "Blog project", - "namespace": "myblog", - "enabled": true, - "configFile": "/path/to/config.yml", - "orgId": "507f1f77bcf86cd799439011", - "teams": ["507f1f77bcf86cd799439012"], - "api": { - "isOnline": true, - "lastReload": "2024-01-16T14:22:00Z", - "namespace": "myblog", - "meta": {}, - "collections": [ - { - "name": "articles", - "defaultLanguage": "german", - "permissions": { - "public": { - "methods": { - "get": { "allow": true }, - "post": { "allow": false }, - "put": { "allow": false }, - "delete": { "allow": false } - }, - "filter": {}, - "validProjections": ["default"] - }, - "user": { - "methods": { - "get": { "allow": true }, - "post": { "allow": true }, - "put": { "allow": true }, - "delete": { "allow": false } - }, - "filter": {}, - "validProjections": ["default"], - "readonlyFields": ["sort"], - "hiddenFields": ["internalNotes"] - } - }, - "projections": { - "default": { "fields": { "title": 1, "content": 1 } } - }, - "fields": [ - { - "name": "title", - "type": "string", - "validator": { "required": true }, - "index": ["text"] - }, - { - "name": "sort", - "type": "number", - "index": ["single"] - }, - { - "name": "tags", - "type": "string[]" - }, - { - "name": "author", - "type": "object", - "subFields": [ - { "name": "name", "type": "string" }, - { "name": "email", "type": "string" } - ] - } - ], - "indexes": [ - { - "name": "textindex", - "key": ["$text:$**"], - "background": true, - "defaultLanguage": "german" - }, - { - "name": "compound_date_status", - "key": ["publishDate", "-status"], - "background": true - } - ], - "search": [ - { - "name": "default", - "mode": "text" - } - ], - "imageFilter": { - "thumbnail": [{"type": "resize", "width": 200, "height": 200}] - }, - "audit": { - "enabled": true, - "actions": ["create", "update", "delete"] - }, - "queryLimits": { - "defaultLimit": 100, - "maxLimit": 1000 - } - } - ], - "assets": [ - { "name": "static", "path": "/var/data/static" } - ], - "jobs": [], - "errors": [] - }, - "yourPermissions": { - "articles": { - "get": { "allow": true }, - "post": { "allow": true }, - "put": { "allow": true }, - "delete": { "allow": false }, - "readonlyFields": ["sort"], - "hiddenFields": ["internalNotes"] - } - } -} -``` - -The response is enriched at runtime with two computed fields: - -**`api` object:** - -| Field | Type | Description | -| -------------- | -------- | -------------------------------------------------------------- | -| `isOnline` | bool | Whether the project is currently loaded and serving requests | -| `lastReload` | datetime | Timestamp of the last successful config reload | -| `namespace` | string | Project namespace (URL segment) | -| `meta` | object | Arbitrary metadata from project config | -| `collections` | array | Full collection configurations (fields, indexes, permissions, projections, search, imageFilter, audit, queryLimits, bulk, meta) | -| `assets` | array | Static asset directories (`name`, `path`) | -| `jobs` | array | Configured background jobs | -| `errors` | array | Config parsing or hook compilation errors | - -**`api.collections[].fields[]` (field-level index info):** - -| Field | Type | Description | -| ------------ | -------- | ------------------------------------------------- | -| `name` | string | Field name | -| `type` | string | Field type (`string`, `number`, `object`, etc.) | -| `index` | string[] | Field-level indexes: `single`, `unique`, `text`, `sparse` | -| `subFields` | array | Nested fields (for `object` type) | -| `validator` | object | Validation rules | -| `readonly` | bool/eval| Read-only flag or eval expression | -| `hidden` | bool/eval| Hidden flag or eval expression | -| `meta` | object | Arbitrary field metadata | - -**`api.collections[].indexes[]` (collection-level indexes):** - -| Field | Type | Description | -| ------------------ | -------- | ---------------------------------------------- | -| `name` | string | Index name | -| `key` | string[] | Index key fields (prefix `-` for descending, `$text:` for text index) | -| `unique` | bool | Whether the index enforces uniqueness | -| `sparse` | bool | Whether the index is sparse | -| `background` | bool | Whether the index was built in background | -| `dropDups` | bool | Whether duplicates were dropped on creation | -| `defaultLanguage` | string | Default language for text indexes | -| `languageOverride` | string | Field name that overrides the language | - -**`yourPermissions` object:** - -Map with collection names as keys. For each collection: - -| Field | Type | Description | -| ---------------- | -------- | ------------------------------------------------ | -| `get` | object | `{"allow": bool}` — read permission | -| `post` | object | `{"allow": bool, "bulk": bool}` — create permission | -| `put` | object | `{"allow": bool, "bulk": bool}` — update permission | -| `delete` | object | `{"allow": bool, "bulk": bool}` — delete permission | -| `readonlyFields` | string[] | Fields the user cannot modify (non-admin only) | -| `hiddenFields` | string[] | Fields hidden from the user (admin: listed as hint; non-admin: stripped from `api.collections`) | - -> **Note for non-admin users:** Hidden fields are removed from the `api.collections[].fields` array in the response. The `hiddenFields` list in `yourPermissions` is only visible to admins as a reference. - -**Auth:** JWT (Admin or assigned User) or Admin-Token with `project.read` permission. - -### Create Project - -```text -POST /api/v1/project -``` - -**Request Body:** - -```json -{ - "name": "My Blog", - "description": "Blog project", - "namespace": "myblog", - "configFile": "/path/to/config.yml", - "enabled": false, - "orgId": "507f1f77bcf86cd799439011", - "teams": ["507f1f77bcf86cd799439012"] -} -``` - -| Field | Type | Required | Description | -| ------------ | -------- | -------- | ------------------------------------------------ | -| `name` | string | Yes | Unique project name | -| `description`| string | Yes | Project description | -| `namespace` | string | No | URL namespace (auto-generated from name if omitted) | -| `configFile` | string | No | Path to the project's YAML config file | -| `enabled` | bool | No | `true` (default) or `false` to disable the project | -| `orgId` | string | No | Owning org ID (determines project visibility) | -| `teams` | string[] | No | Team IDs assigned to this project (grants working rights) | - -After creation, the project config is automatically loaded and the collections are initialized. - -**Response (200):** Returns the created project (same format as Get Project, including enriched `api` and `yourPermissions`). - -**Auth:** Admin only, or Admin-Token with `project.create` permission. - -### Update Project - -```text -PUT /api/v1/project/:id -``` - -Partial update — only send fields to change. Triggers a project reload (re-reads config, recompiles hooks, restarts jobs). - -To disable a project: `PUT /api/v1/project/:id` with `{"enabled": false}`. To re-enable: `{"enabled": true}`. - -**Response (200):** Returns the updated project (same format as Get Project). - -**Auth:** Admin only, or Admin-Token with `project.update` permission. - -### Delete Project - -```text -DELETE /api/v1/project/:id -``` - -**Response (200):** - -```json -{ - "message": "ok" -} -``` - -**Auth:** Admin only, or Admin-Token with `project.delete` permission. - -### Project Enable/Disable - -Projects can be taken offline without deleting them by setting `enabled` to `false`: - -```text -PUT /api/v1/project/:id -Content-Type: application/json - -{ "enabled": false } -``` - -**Behavior:** -- `enabled` is `null` or `true` → project is **online** (normal operation). This is the default — existing projects without the field behave as before (backward compatible). -- `enabled` is `false` → project is **offline**. The project is completely removed from the router. All API requests to its namespace (collections, assets, admin/reload) return **404**. The project record remains in the database and is visible in `GET /project` / `GET /project/:id`. The `api` field in GET responses will be empty. - -Re-enabling: - -```text -PUT /api/v1/project/:id -Content-Type: application/json - -{ "enabled": true } -``` - -This triggers a full project reload (same as initial setup). - ---- - -## System Endpoint Lookup - -All system model GET endpoints (`user`, `project`, `org`, `team`) support the `?lookup=` parameter to resolve foreign-key references inline — the same feature available on collection endpoints, with system-specific restrictions. - -### Syntax - -The syntax is identical to [collection lookups](#list-documents): - -```bash -# Colon-separated -?lookup=orgId:org - -# Multiple lookups -?lookup=orgId:org,primaryOrgId:org - -# JSON syntax -?lookup={"field":"orgId","collection":"org"} - -# Repeated / bracket notation -?lookup=orgId:org&lookup=primaryOrgId:org -?lookup[]=orgId:org&lookup[]=primaryOrgId:org -``` - -### Restrictions (vs. collection lookups) - -| Rule | Collection Endpoints | System Endpoints | -|------|---------------------|------------------| -| Target collection | Any collection in the project | Only `user`, `project`, `org`, `team` | -| Collection param | Optional (auto-detected from field name) | **Required** (no auto-detection) | -| Projection param | Supported (named projections) | Not applicable | -| Hooks | Target hooks execute | No hooks | -| Fast path ($lookup) | Yes, when eligible | No (always application-level) | - -### Allowed target collections - -Only the four system models can be lookup targets: - -- `user` — e.g. `?lookup=createdBy:user` -- `project` — e.g. `?lookup=projectId:project` -- `org` — e.g. `?lookup=orgId:org` -- `team` — e.g. `?lookup=teamId:team` - -Any other target collection returns `400`. - -### Security - -- **Authorization scoping:** Each lookup target is filtered through the target model's `ValidateFilter`. A non-admin user can only see lookup results they would normally be able to access via the target's own GET endpoint. For example, an orgA member looking up users will only see users who share at least one org. -- **Sensitive field stripping:** Results are serialized through each model's `MarshalJSON`, which strips sensitive fields. Most notably, the `password` field is **never** included in user lookup results. -- **Field name validation:** Field names and `as` names are validated against `^[a-zA-Z0-9_]+(\.[a-zA-Z0-9_]+)*$` — no injection possible. - -### Examples - -**Team with resolved org:** - -```bash -GET /api/v1/team/507f1f77bcf86cd799439014?lookup=orgId:org -``` - -```json -{ - "id": "507f1f77bcf86cd799439014", - "name": "Frontend Team", - "orgId": "507f1f77bcf86cd799439013", - "_lookup": { - "orgId": { - "id": "507f1f77bcf86cd799439013", - "name": "ACME Corp" - } - } -} -``` - -**Project list with org lookup:** - -```bash -GET /api/v1/project?lookup=orgId:org -``` - -```json -[ - { - "id": "abc123", - "name": "My Blog", - "orgId": "507f1f77bcf86cd799439013", - "_lookup": { - "orgId": { - "id": "507f1f77bcf86cd799439013", - "name": "ACME Corp" - } - } - } -] -``` - -**User with resolved primary org:** - -```bash -GET /api/v1/user/507f1f77bcf86cd799439011?lookup=primaryOrgId:org -``` - -```json -{ - "id": "507f1f77bcf86cd799439011", - "username": "alice", - "primaryOrgId": "507f1f77bcf86cd799439013", - "_lookup": { - "primaryOrgId": { - "id": "507f1f77bcf86cd799439013", - "name": "ACME Corp" - } - } -} -``` - -**Error — missing collection:** - -```bash -GET /api/v1/project?lookup=orgId -# → 400: lookup target collection is required for system endpoints -``` - -**Error — forbidden collection:** - -```bash -GET /api/v1/project?lookup=orgId:posts -# → 400: lookup target collection not allowed: posts -``` - -### Array-of-Objects (Nested Lookup) - -For fields that contain an **array of objects** (e.g. `orgs: [{orgId: "...", role: "..."}]`), dot-notation traverses into each array element and resolves the foreign key per element. - -**Syntax:** - -```bash -# Resolve orgId inside each orgs[] element -?lookup=orgs.orgId:org - -# Resolve teamId inside each teams[] element -?lookup=teams.teamId:team - -# Combine flat + nested lookups -?lookup[]=primaryOrgId:org&lookup[]=orgs.orgId:org -``` - -**Result structure — per-element `_lookup`:** - -Each array element receives its own `_lookup` sub-object with the resolved data: - -```bash -GET /api/v1/user/507f1f77bcf86cd799439011?lookup=orgs.orgId:org -``` - -```json -{ - "id": "507f1f77bcf86cd799439011", - "username": "alice", - "orgs": [ - { - "orgId": "507f1f77bcf86cd799439013", - "role": "admin", - "_lookup": { - "orgId": { - "id": "507f1f77bcf86cd799439013", - "name": "ACME Corp" - } - } - }, - { - "orgId": "507f1f77bcf86cd799439014", - "role": "member", - "_lookup": { - "orgId": { - "id": "507f1f77bcf86cd799439014", - "name": "Other Inc" - } - } - } - ] -} -``` - -**Combined flat + nested lookup:** - -```bash -GET /api/v1/user/507f1f77bcf86cd799439011?lookup[]=primaryOrgId:org&lookup[]=orgs.orgId:org -``` - -```json -{ - "id": "507f1f77bcf86cd799439011", - "username": "alice", - "primaryOrgId": "507f1f77bcf86cd799439013", - "orgs": [ - { - "orgId": "507f1f77bcf86cd799439013", - "role": "admin", - "_lookup": { - "orgId": { "id": "507f1f77bcf86cd799439013", "name": "ACME Corp" } - } - } - ], - "_lookup": { - "primaryOrgId": { - "id": "507f1f77bcf86cd799439013", - "name": "ACME Corp" - } - } -} -``` - -**Typical use cases:** -- `?lookup=orgs.orgId:org` — Resolve org names for each membership entry -- `?lookup=teams.teamId:team` — Resolve team names for each team membership -- Works on both list and GetOne endpoints -- Security: Same authorization scoping and field stripping applies (ValidateFilter + MarshalJSON) - ---- - -## Collection Endpoints - -Dynamic endpoints based on project configuration. The `:project` parameter is the project's namespace. - -### List Documents - -```text -GET /api/v1/_/:project/:collection -``` - -**Query Parameters:** - -| Parameter | Type | Default | Description | -| ------------ | ------------- | ------- | -------------------------------------------------------------------- | -| `filter` | JSON string | — | MongoDB filter query (validated against whitelist) | -| `q` | string | — | Search term. Triggers search filter (text or regex, see [Search](04-collections.md#search-q-parameter)) | -| `qName` | string | — | Named search config to use with `q` (default: first config) | -| `qExplain` | `1` or `true` | — | Include `_search.details` score breakdowns for vector and combined list responses | -| `limit` | int | 10000 | Maximum number of results (capped at `maxLimit`, see below) | -| `offset` | int | 0 | Number of results to skip | -| `sort` | string / JSON | — | Sort fields (see below) | -| `sort[]` | string[] | — | Alternative: multiple sort parameters (bracket notation) | -| `count` | `1` or `true` | — | Return count in `X-Results-Count` header and `Content-Range` | -| `projection` | string | — | Named projection from collection config | -| `select` | string / JSON | — | Field selection (see below) | -| `select[]` | string[] | — | Alternative: multiple select parameters | -| `lookup` | string / JSON | — | Resolve foreign-key references inline (see below) | -| `lookup[]` | string[] | — | Alternative: multiple lookup parameters (bracket notation) | -| `aggregate` | string / JSON | — | Compute reverse aggregates from a target collection (see below) | -| `aggregate[]` | string[] | — | Alternative: multiple aggregate parameters (bracket notation) | - -**Query Limits:** - -All list endpoints enforce query result limits to prevent accidental full-table scans: - -- **Default limit:** 10,000 results (if no `limit` parameter is provided) -- **Maximum limit:** 10,000 results (requests above this return a 400 error) -- Both values are configurable per collection via `queryLimits` in the collection YAML (see [Collections](04-collections.md#query-limits)). -- To retrieve more results, use pagination with `offset`. - -**Sorting (`sort`):** - -The `sort` parameter supports multiple formats. Use `-` prefix for descending order: - -```bash -# Comma-separated (ascending by default, "-" prefix for descending) -?sort=title,-created - -# Repeated params -?sort=title&sort=-created - -# Bracket notation -?sort[]=title&sort[]=-created - -# MongoDB JSON syntax (1 = ascending, -1 = descending) -?sort={"title":1,"created":-1} - -# $meta textScore sort (for $text search results) -?sort={"score":{"$meta":"textScore"},"title":1} -``` - -Key order in the JSON format determines sort priority (primary sort first). - -When using `$text` search queries (via `filter`), you can sort by text relevance using the -`{"$meta":"textScore"}` syntax in the JSON sort format. This assigns a text-match score to -the given field name and sorts by it. Example: - -```bash -?filter={"$text":{"$search":"hello"}}&sort={"score":{"$meta":"textScore"}} -``` - -The `$textScore:` encoding (`$textScore:score`) is used internally throughout the pipeline -(controller → hooks → ReadCollection → MongoDB) so the sort format stays `[]string` and is -fully backwards-compatible. Hooks receive and can modify these entries like any other sort field. - -**Field Selection (`select`):** - -The `select` parameter allows clients to restrict which fields are returned. It supports multiple formats: - -```bash -# Comma-separated field list (include mode) -?select=title,author,sort - -# Repeated parameter -?select=title&select=author - -# Bracket notation -?select[]=title&select[]=author - -# MongoDB JSON syntax — include -?select={"title":1,"author":1} - -# MongoDB JSON syntax — exclude -?select={"secret":0} - -# Exclude _id (MongoDB convention, works with include mode) -?select={"title":1,"_id":0} -``` - -**Select + Projection enforcement:** The select parameter can only **restrict** the output further, never widen it beyond what the permission-based projection allows. For security, non-existent fields and disallowed fields return the same generic `400` error message. - -**Behavior with and without `?projection=`:** - -- **`?select=` with `?projection=`:** The select is merged with the explicit projection. The select can only narrow the projection, never widen it. Requesting a field excluded by the projection returns `400`. -- **`?select=` without `?projection=`:** The select is validated against the **union** of all projections the current role is allowed to use (`validProjections`). A field is allowed if **any** single valid projection would return it. The select becomes the MongoDB selector directly — the client gets exactly the requested fields. This is safe because the user could access all those fields via separate requests with different projections anyway. -- **No `?select=`:** Behavior is unchanged — the first valid projection (or the default) is applied automatically. - -**Rules:** -- Comma-separated / repeated / bracket formats always produce include-mode selections (`field: 1`) -- JSON format supports both include (`1`) and exclude (`0`), but mixing is not allowed (MongoDB rule) -- `_id` can always be excluded via `{"_id": 0}` even in include mode -- Fields are validated against the collection schema — unknown fields are rejected when `strictFields: true` is set (project, collection, or field level; default: false = allow unknown fields) -- Admins have no projection restrictions and can select any field - -**Response Headers (when `count=1`):** - -```text -X-Results-Count: 42 -Content-Range: entry 0-41/42 -``` - -**Filter Examples:** - -```bash -# Simple equality -?filter={"published":true} - -# Comparison operators -?filter={"age":{"$gte":18}} - -# Logical operators -?filter={"$or":[{"status":"active"},{"featured":true}]} - -# Text search -?filter={"$text":{"$search":"hello world"}} - -# Regex -?filter={"title":{"$regex":"^Hello","$options":"i"}} -``` - -**Lookup (`lookup`):** - -The `lookup` parameter resolves foreign-key references inline, embedding the referenced document(s) directly into the response. This avoids N+1 client-side queries. - -**Basic syntax (colon-separated):** - -```bash -# Resolve authorId → look up in "authors" collection -?lookup=authorId:authors - -# Auto-detect target collection (authorId → authors, categoryId → categories) -?lookup=authorId - -# Multiple lookups (comma-separated) -?lookup=authorId:authors,categoryId:categories - -# Multiple lookups (repeated parameter) -?lookup=authorId:authors&lookup=categoryId:categories - -# Bracket notation -?lookup[]=authorId:authors&lookup[]=categoryId:categories -``` - -**Full colon-separated format:** - -``` -?lookup=field:collection:foreignField:projection:select:as -``` - -| Segment | Default | Description | -| --------------- | -------------------- | ------------------------------------------------------------ | -| `field` | *(required)* | Source field containing the reference value(s) | -| `collection` | *(auto-detected)* | Target collection to look up in | -| `foreignField` | `_id` | Field in target collection to match against | -| `projection` | *(role default)* | Named projection to apply on target results | -| `select` | — | Additional field selection (pipe-separated: `name\|email`) | -| `as` | `_lookup.{field}` | Destination field name in the response | - -Empty segments use their defaults: `authorId:authors:::name|email` uses `_id` as foreignField and the role's default projection. - -**JSON syntax:** - -```bash -# Single lookup -?lookup={"field":"authorId","collection":"authors"} - -# Array of lookups -?lookup=[{"field":"authorId","collection":"authors"},{"field":"categoryId","collection":"categories"}] - -# With field selection (JSON map — values must be 0 or 1) -?lookup={"field":"authorId","collection":"authors","select":{"name":1,"email":1}} - -# With field selection (string — comma-separated) -?lookup={"field":"authorId","collection":"authors","select":"name,email"} -``` - -JSON properties: `field` (required), `collection`, `foreignField`, `projection`, `select`, `as`. - -**Response format:** - -Resolved documents are placed under a `_lookup` namespace by default: - -```json -{ - "id": "abc123", - "title": "My Post", - "authorId": "def456", - "_lookup": { - "authorId": { - "id": "def456", - "name": "Alice", - "email": "alice@example.com" - } - } -} -``` - -For array fields (e.g. `string[]`), the resolved value is an array of documents: - -```json -{ - "tagIds": ["t1", "t2"], - "_lookup": { - "tagIds": [ - {"id": "t1", "name": "Go"}, - {"id": "t2", "name": "MongoDB"} - ] - } -} -``` - -If a referenced document is not found (deleted or filtered by permissions), its position in the array is `null`. - -**Auto-detection:** - -When no target collection is specified, the server infers it from the field name: - -| Field Name | Detected Collection | -| ----------------- | ------------------- | -| `userId` | `users` | -| `categoryId` | `categories` | -| `user_id` | `users` | -| `productIds` | `products` | -| `items.productId` | `products` | - -The detection tries plurals (`+s`, `+es`, `y→ies`) and falls back to exact match. If auto-detection fails, a `400` error is returned. - -**Nested fields (dot notation):** - -Lookups support dot-notation for fields inside nested objects or object arrays: - -```bash -# Look up productId inside each element of the "items" array -?lookup=items.productId:products -``` - -The result is embedded per array element with a `_lookup` sub-object: - -```json -{ - "items": [ - { - "productId": "p1", - "qty": 2, - "_lookup": { - "productId": {"id": "p1", "name": "Widget"} - } - }, - { - "productId": "p2", - "qty": 1, - "_lookup": { - "productId": {"id": "p2", "name": "Gadget"} - } - } - ] -} -``` - -**Security & permissions:** - -- The user must have `get` permission on the **target** collection. Otherwise → `403`. -- Projection restrictions (`validProjections`) of the target collection are enforced: the role's default projection is applied automatically. An explicit `projection` segment must be in the role's `validProjections` list. -- The `select` parameter can only **narrow** the projection, never widen it (same rules as the regular `?select=`). -- Pre-filters (`permissions.filter`) on the target collection are applied — results the user cannot normally see are not resolved. -- Target collection hooks (`get.read`, `get.return`) are executed for each lookup target, including `pipelineMod` from hooks. -- `foreignField` and `as` are validated against `^[a-zA-Z0-9_]+(\.[a-zA-Z0-9_]+)*$`. -- `select` values in JSON map format are restricted to `0` or `1` (no aggregation expressions). - -**Performance — Fast path vs. safe path:** - -The server automatically chooses between two execution strategies: - -| Strategy | When | How | -| ----------- | --------------------------------------------------------- | --------------------------------------------------------- | -| **Fast path** | `foreignField=_id`, target has no hooks, DB supports `$lookup` | Single MongoDB aggregation pipeline with `$lookup` stages | -| **Safe path** | Any other case | Separate queries per target collection (application-level) | - -The fast path is significantly more efficient (one round-trip to MongoDB), while the safe path ensures full hook execution and supports custom `foreignField` values. The choice is transparent to the client — the response format is identical. - -Dot-notation fields (e.g. `items.productId`) are fully supported on the fast path. If the first segment is an array of objects, the pipeline uses `$map`/`$mergeObjects`/`$filter` to embed lookup results per array element. If it's a plain nested object, the pipeline falls back to simple `$arrayElemAt`. The detection happens at runtime via `$isArray`. - -**Aggregate (`aggregate`):** - -The `aggregate` parameter computes reverse aggregates against another collection and stores the result under `_aggregate`. Unlike `lookup`, it does not embed target documents. Instead, it returns scalar values such as counts, existence checks, sums, averages, minima, or maxima. - -Typical use cases: - -- Count posts per author -- Check whether an author has any published posts -- Sum sales totals for the current document's id -- Compute per-item counts for nested array fields such as `items.productId` - -Conceptually, each aggregate spec works like this: - -1. Read a source value from the current document via `field` (default: `_id`) -2. Query the target collection -3. Match target documents where `foreignField == sourceValue` -4. Apply the chosen operation (`count`, `exists`, `sum`, `avg`, `min`, `max`) -5. Write the result to `_aggregate.{key}` - -**Basic syntax (colon-separated short form):** - -```bash -# Count posts whose authorId matches the current document's _id -?aggregate=posts:authorId - -# Exists instead of count -?aggregate=posts:authorId:exists - -# Sum a numeric valueField -?aggregate=posts:authorId:sum:price - -# Custom result key -?aggregate=posts:authorId:sum:price:postRevenue - -# Multiple aggregates (comma-separated) -?aggregate=posts:authorId,posts:authorId:exists - -# Multiple aggregates (repeated) -?aggregate=posts:authorId&aggregate=sales:authorId:sum:publicAmount - -# Bracket notation -?aggregate[]=posts:authorId&aggregate[]=sales:authorId:sum:publicAmount -``` - -**Full colon-separated format:** - -```text -?aggregate=collection:foreignField:op:valueField:as -``` - -| Segment | Default | Description | -| -------------- | -------------------------- | ------------------------------------------------------------------ | -| `collection` | *(required)* | Target collection to aggregate over | -| `foreignField` | *(required)* | Field in the target collection matched against the source value | -| `op` | `count` | Aggregate operation: `count`, `exists`, `sum`, `avg`, `min`, `max` | -| `valueField` | — | Numeric field used by `sum`, `avg`, `min`, `max` | -| `as` | derived key in `_aggregate` | Optional custom result key | - -Important limitations of the short form: - -- The source field is always the current document's root `_id` -- No target-side `filter` can be expressed in the short form -- No collection auto-detection exists; `collection` must be given explicitly - -If you need a custom source field, nested field, or target-side filter, use JSON syntax. - -**JSON syntax:** - -```bash -# Single aggregate with defaults: field=_id, op=count -?aggregate={"collection":"posts","foreignField":"authorId"} - -# Numeric aggregate -?aggregate={"collection":"posts","foreignField":"authorId","op":"sum","valueField":"price"} - -# Aggregate by a custom source field instead of the current _id -?aggregate={"field":"authorId","collection":"sales","foreignField":"ownerId","op":"sum","valueField":"gross"} - -# Aggregate with an additional target-side filter -?aggregate={"collection":"posts","foreignField":"authorId","filter":{"status":"published"}} - -# Multiple aggregates in one JSON array -?aggregate=[ - {"collection":"posts","foreignField":"authorId","op":"count"}, - {"collection":"posts","foreignField":"authorId","op":"exists","as":"hasPosts"} -] -``` - -JSON properties: - -| Property | Default | Description | -| -------------- | -------------------------- | ------------------------------------------------------------------ | -| `field` | `_id` | Source field path on the current document | -| `collection` | *(required in practice)* | Target collection | -| `foreignField` | *(required)* | Target field matched against the source value | -| `op` | `count` | Aggregate operation | -| `valueField` | — | Numeric target field for `sum`, `avg`, `min`, `max` | -| `filter` | — | Additional filter applied to target documents before aggregation | -| `as` | derived key in `_aggregate` | Optional custom result key | - -Notes: - -- `field`, `foreignField`, and `valueField` accept `id` as an alias for root `_id` -- String syntax and JSON syntax can both be repeated via `aggregate` or `aggregate[]` -- Invalid or duplicate derived result keys return `400` - -**Response format:** - -Aggregate results are stored under `_aggregate` on the object where the source field lives. - -Top-level example: - -```json -{ - "id": "a1", - "name": "Alice", - "_aggregate": { - "postsCount": 2, - "postsExists": true, - "postsPriceSum": 30, - "postsPriceAvg": 15, - "postsPriceMin": 10, - "postsPriceMax": 20 - } -} -``` - -By default, the result key is derived from the target collection, optional `valueField`, and operation: - -- `postsCount` -- `postsExists` -- `postsPriceSum` -- `salesPublicAmountSum` - -Use `as` to override the key: - -```bash -?aggregate={"collection":"posts","foreignField":"authorId","op":"count","as":"publishedPostCount"} -``` - -Then the response contains: - -```json -{ - "_aggregate": { - "publishedPostCount": 2 - } -} -``` - -**Empty-result defaults:** - -When no target documents match, the server returns deterministic empty values: - -| Operation | Empty result | -| --------- | ------------ | -| `count` | `0` | -| `exists` | `false` | -| `sum` | `0` | -| `avg` | `null` | -| `min` | `null` | -| `max` | `null` | - -**Nested fields and arrays:** - -JSON syntax supports aggregates on nested source fields: - -```bash -# Aggregate per nested object field -?aggregate={"field":"profile.managerId","collection":"users","foreignField":"_id","op":"exists","as":"managerExists"} - -# Aggregate per array element -?aggregate={"field":"items.productId","collection":"reviews","foreignField":"productId","op":"count","as":"reviewCount"} -``` - -For nested array/object fields, the aggregate result is written at the same nesting level as the source field. Example response for `field=items.productId`: - -```json -{ - "items": [ - { - "productId": "p1", - "qty": 2, - "_aggregate": { - "reviewCount": 5 - } - }, - { - "productId": "p2", - "qty": 1, - "_aggregate": { - "reviewCount": 1 - } - } - ] -} -``` - -If the source field itself contains multiple values, operations combine them as follows: - -- `count`: sums the counts across all referenced source values -- `exists`: becomes `true` as soon as any referenced source value has a match -- `sum`: sums all numeric contributions across all referenced source values -- `avg`: computes a weighted average across all matched numeric values -- `min` / `max`: compute the minimum or maximum across all matched numeric values - -**Target-side filtering:** - -The optional `filter` in JSON syntax is applied to target documents after the join condition and after permission pre-filters. This lets you ask questions such as: - -```bash -# Count only published posts per author -?aggregate={"collection":"posts","foreignField":"authorId","filter":{"status":"published"}} - -# Sum only visible revenue rows of a certain type -?aggregate={"collection":"sales","foreignField":"authorId","op":"sum","valueField":"publicAmount","filter":{"kind":"invoice"}} -``` - -The filter uses the same operator rules as normal `filter` queries. Field-level readability rules still apply. - -**Security, permissions, and hooks:** - -- The caller must have `get` permission on the target collection. Otherwise the request fails with `403`. -- Target collection pre-filters (`permissions.filter`) are always applied. -- The target `foreignField` must exist and be readable for the current role. -- Any field referenced inside the aggregate `filter` must be readable. -- For `sum`, `avg`, `min`, and `max`, `valueField` must exist, be numeric, and be readable. -- Hidden fields and eval-hidden rules on the target collection are respected. -- Target hooks `get.read` and `get.return` are executed, so aggregate results reflect the same read-time restrictions as normal GET requests. - -This means `aggregate` does not bypass collection permissions, pre-filters, projections, hidden fields, or read hooks. - -**Behavior and performance:** - -- The current implementation executes aggregates in the application layer, not via a MongoDB `$lookup` response embedding stage. -- Compatible aggregate specs are grouped by source field, target collection, target `foreignField`, and effective filter to avoid redundant target reads. -- Lookup values are processed in batches, so multiple rows can share the same grouped aggregate query. -- The response shape is stable regardless of how many grouped queries were required internally. - -**Common examples:** - -```bash -# Count posts per author -GET /api/v1/_/myproject/authors?aggregate=posts:authorId - -# Count and exists in one request -GET /api/v1/_/myproject/authors?aggregate=posts:authorId&aggregate=posts:authorId:exists - -# Sum post prices per author -GET /api/v1/_/myproject/authors?aggregate=posts:authorId:sum:price - -# Count only published posts per author -GET /api/v1/_/myproject/authors?aggregate={"collection":"posts","foreignField":"authorId","filter":{"status":"published"}} - -# Per-item aggregate for nested array elements -GET /api/v1/_/myproject/orders?aggregate={"field":"items.productId","collection":"reviews","foreignField":"productId","op":"count","as":"reviewCount"} -``` - ---- - -**Allowed Filter Operators:** See [Collections](04-collections.md#filter-operators). - -**Example:** - -```bash -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles?limit=10&offset=0&sort=-publishDate&filter={\"published\":true}" -``` - -**Hook Pipeline:** `request` → `read` → [DB query] → `return` - ---- - -### Collection Queue Status - -```text -GET /api/v1/_/:project/:collection/_queue -``` - -Returns the count of documents currently pending vector or search enrichment via the asynchronous `autoRegenerate` queue (used with the Outbox Pattern). - -**Response:** - -```json -{ - "queueLength": 42 -} -``` - ---- - -### Count Documents - -```text -GET /api/v1/_/:project/:collection/_count -``` - -Returns the count of documents matching the given filter, without loading the actual documents. -This is more efficient than using `?count=1` on the list endpoint when you only need the count. - -**Query Parameters:** - -| Parameter | Type | Default | Description | -| ---------- | ----------- | ------- | ---------------------------------------------------------------------------------------------------- | -| `filter` | JSON string | — | MongoDB filter query (validated against whitelist) | -| `q` | string | — | Full-text or regex search term (see [Search](04-collections.md#search-q-parameter)) | -| `qName` | string | — | Name of a configured search entry to use instead of automatic fallback | -| `estimate` | string | — | Set to `true` or `1` to use MongoDB's `EstimatedDocumentCount` (O(1), ignores filter if present) | - -When `estimate=true` is set **and** no filter is active (neither via query parameter, pre-filter, nor -hook), the endpoint uses MongoDB's `EstimatedDocumentCount` which reads collection metadata and returns -almost instantly. If a filter is present, the parameter is silently ignored and an exact count is returned. - -**Response (200):** - -```json -{ - "count": 42 -} -``` - -**Permissions:** Uses the same `get` permission as the list endpoint. Pre-filters (role-based) are applied. - -**Hook Pipeline:** `request` → `read` → [DB count] - -The `get.read` hook is executed before counting, so any filter modifications made by the hook -are applied to the count query. The `get.return` hook is **not** executed (no documents to transform). - -**Example:** - -```bash -# Count all documents -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles/_count" - -# Count with filter -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles/_count?filter={\"published\":true}" - -# Fast estimated count (reads collection metadata, O(1)) -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles/_count?estimate=true" -``` - -> **Note:** The existing `?count=1` query parameter on the list endpoint continues to work -> and returns the count in `X-Results-Count` / `Content-Range` headers alongside the documents. -> Use `/_count` when you only need the count without loading documents. - ---- - -### Faceted Aggregation - -```text -GET /api/v1/_/:project/:collection/_facet -``` - -Returns distinct values with document counts for one or more fields. Designed for building -faceted filters (e.g. filter sidebar with counts) in UIs. Uses MongoDB's `$facet` aggregation -for efficient single-roundtrip execution across multiple fields. - -**Query Parameters:** - -| Parameter | Type | Default | Description | -| --------- | ----------- | ------- | --------------------------------------------------------------------------------------------------- | -| `facet` | string list | — | **Required.** Fields to aggregate. Comma-separated, repeated, or bracket notation (see below) | -| `filter` | JSON string | — | MongoDB filter query applied before faceting (validated against whitelist) | -| `q` | string | — | Full-text or regex search term (see [Search](04-collections.md#search-q-parameter)) | -| `qName` | string | — | Name of a configured search entry to use instead of automatic fallback | -| `limit` | int | `100` | Maximum number of distinct values returned per field | -| `sort` | string | `count` | Sort mode: `count` (by count descending, default) or `value` (alphabetical by value ascending) | - -**Facet parameter formats:** - -```bash -# Comma-separated -?facet=status,category - -# Repeated parameter -?facet=status&facet=category - -# Bracket notation -?facet[]=status&facet[]=category -``` - -Dot-notation for nested fields is supported: `?facet=address.city` - -**Response (200):** - -```json -{ - "facets": { - "status": { - "values": [ - { "value": "published", "count": 42 }, - { "value": "draft", "count": 15 }, - { "value": null, "count": 3 } - ], - "total": 3 - }, - "category": { - "values": [ - { "value": "news", "count": 30 }, - { "value": "blog", "count": 20 }, - { "value": "tutorial", "count": 10 } - ], - "total": 3 - } - } -} -``` - -- `values[].value`: The distinct field value (can be `null` for documents where the field is missing/null) -- `values[].count`: Number of documents with this value -- `total`: Number of unique values returned for this field (respects `limit`) - -**Permissions:** Uses the same `get` permission as the list endpoint. Pre-filters (role-based) are applied. -Faceting on hidden fields (collection-level, permission-set-level, or field-level `hidden: true`) is blocked with a 400 error. - -**Hook Pipeline:** `request` → `read` → [DB facet] - -The `get.read` hook is executed before faceting, so any filter modifications are applied. -The `get.return` hook is **not** executed (no documents to transform). - -**Examples:** - -```bash -# Single field facet -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles/_facet?facet=status" - -# Multiple fields with filter -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles/_facet?facet=status,category&filter={\"published\":true}" - -# Alphabetical sort with limit -curl -H "X-Auth-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myblog/articles/_facet?facet=tags&sort=value&limit=20" -``` - ---- - -### Get Document - -```text -GET /api/v1/_/:project/:collection/:id -``` - -Returns a single document by ObjectID. - -**Response (200):** - -```json -{ - "id": "507f1f77bcf86cd799439011", - "insertTime": "2024-01-15T10:30:00Z", - "updateTime": "2024-01-16T14:22:00Z", - "title": "Hello World", - "published": true -} -``` - -**Response (404):** - -```json -{ - "error": "not found", - "errorCode": "20240115103000-a1b2" -} -``` - ---- - -### Get Single Field - -```text -GET /api/v1/_/:project/:collection/:id/:field -``` - -Returns only the value of a specific field from a document. - -**Example:** - -```bash -# Returns just the title string -curl http://localhost:8080/api/v1/_/myblog/articles/507f1f77bcf86cd799439011/title -``` - ---- - -### Get File - -```text -GET /api/v1/_/:project/:collection/:id/:field/*path -``` - -Serves a file attached to a document field. - -**Query Parameters:** - -| Parameter | Type | Description | -| ---------- | ------ | -------------------------------------------------------- | -| `filter` | string | Named image filter preset from collection config | -| `no-cache` | `1` | Force regeneration of filtered image | - -**Example:** - -```bash -# Get original file -curl http://localhost:8080/api/v1/_/myblog/articles/507f.../image/photo.jpg - -# Get filtered version (thumbnail preset) -curl http://localhost:8080/api/v1/_/myblog/articles/507f.../image/photo.jpg?filter=thumbnail -``` - -**Hook Pipeline:** `file` (can intercept/modify the response) - ---- - -### Create Document - -```text -POST /api/v1/_/:project/:collection -``` - -**Request Body (JSON):** - -```json -{ - "title": "New Article", - "content": "Article content...", - "published": false, - "image": { - "src": "data:image/jpeg;base64,/9j/4AAQ...", - "path": "photo.jpg" - } -} -``` - -File uploads are embedded as Base64 data URIs in the JSON body. See [File Upload & Images](08-file-upload-images.md). - -**Response (200):** - -```json -{ - "id": "507f1f77bcf86cd799439011", - "insertTime": "2024-01-15T10:30:00Z", - "title": "New Article", - "content": "Article content...", - "published": false, - "image": { - "src": "image/photo.jpg", - "type": "image/jpeg" - } -} -``` - -**Custom ID:** Include `"id": "hexstring"` in the body to use a specific ObjectID. - -**Hook Pipeline:** `bind` → `validate` → `create` → [DB insert] → `return` - ---- - -### Bulk Create Documents - -```text -POST /api/v1/_/:project/:collection -Content-Type: application/json -Body: JSON array -``` - -Creates multiple documents in a single request. The server automatically detects bulk mode when the request body is a **JSON array** instead of a JSON object. Uses the same route as single-document POST. - -**Requires:** `post: {allow: true, bulk: true}` in the permission set. A boolean `post: true` only allows single-document POST. - -**Request Body (JSON array):** - -```json -[ - {"title": "Article 1", "sort": 1}, - {"title": "Article 2", "sort": 2}, - {"title": "Article 3", "sort": 3} -] -``` - -File uploads are supported per document via Base64 data URIs, same as single POST. - -**Query Parameters:** - -| Parameter | Required | Description | -| --------- | -------- | ----------- | -| `dryRun` | No | `1` or `true` — returns document count without creating | - -**Response (200):** - -```json -{ - "created": 3, - "ids": ["507f1f77bcf86cd799439011", "507f1f77bcf86cd799439012", "507f1f77bcf86cd799439013"] -} -``` - -**Response (200, dry-run):** - -```json -{ - "dryRun": true, - "matched": 3 -} -``` - -**Response (200, with validation errors in per-document mode):** - -```json -{ - "created": 2, - "ids": ["507f1f77bcf86cd799439011", "507f1f77bcf86cd799439012"], - "failed": 1, - "errors": [ - {"index": 2, "error": "validation failed", "validationErrors": {"title": {"message": "required"}}} - ] -} -``` - -**Execution Modes:** - -1. **Per-document (default):** Without a `bulkCreate` hook step and without `bulk.optimize.create`, each document is created individually with the full hook pipeline (`bind` → `validate` → field validation → `create` → file upload → DB insert → audit → `return`). Failed documents are collected in `errors`; successful documents continue. - -2. **Optimised (with `bulkCreate` hook or `bulk.optimize.create: true`):** The hook runs once with the full data array (if defined), then all documents are validated and inserted in a single `InsertMany` DB call. **Per-document hooks (`bind`, `validate`, `create`, `return`) are completely skipped.** If any document fails validation, the entire batch is aborted. - -> **Warning:** The optimised path skips per-document hooks. Use only when no per-document business logic is needed, or handle it in the `bulkCreate` hook. - -> **Configuration without hook file:** -> -> ```yaml -> collections: -> - name: products -> bulk: -> optimize: -> create: true -> ``` -> -> This activates the optimised path without needing a hook file. - ---- - -### Update Document - -```text -PUT /api/v1/_/:project/:collection/:id -``` - -Partial update — only send fields to change. The `_id`/`id`, `insertTime`, and `updateTime` fields are automatically excluded. - -**Request Body:** - -```json -{ - "title": "Updated Title", - "published": true -} -``` - -**Response (200):** Returns the updated document. - -**Response (404):** - -```json -{ - "error": "document not found" -} -``` - -**Hook Pipeline:** `bind` → `validate` → `update` → [DB update] → `return` - -#### Computed Updates (Field-Level Operators) - -Both single-document PUT and bulk PUT support field-level update operators for computed/relative updates: - -| Operator | Description | Example | -| -------- | ----------- | ------- | -| `$inc` | Increment/decrement by value | `{"sort": {"$inc": 5}}` | -| `$mul` | Multiply by factor | `{"price": {"$mul": 1.1}}` | -| `$unset` | Remove field (set to null) | `{"old_field": {"$unset": true}}` | -| `$set` | Explicit set (same as plain value) | `{"title": {"$set": "new"}}` | - -**Example — increase price by 10%:** - -```json -{ - "price": { "$mul": 1.1 }, - "title": "Also update title as plain value" -} -``` - -Operators and plain values can be mixed. Operators are resolved against the current document value before hooks and validation run, so hooks always see concrete values. Protected fields (`id`, `_id`, `insertTime`, `updateTime`) are still excluded automatically. - ---- - -### Delete Document - -```text -DELETE /api/v1/_/:project/:collection/:id -``` - -Deletes the document and all associated files from the filesystem. - -**Response (200):** - -```json -{ - "message": "ok" -} -``` - -**Hook Pipeline:** `delete` → [DB delete] → [file cleanup] → `return` - ---- - -### Bulk Update Documents - -```text -PUT /api/v1/_/:project/:collection?filter={...} -``` - -Updates all documents matching the filter. The filter is a JSON-encoded MongoDB query passed as query parameter. Same permission system and pre-filters as single-document updates. - -**Query Parameters:** - -| Parameter | Required | Description | -| --------- | -------- | ----------- | -| `filter` | Yes | JSON-encoded MongoDB filter query | -| `dryRun` | No | `1` or `true` — returns match count without updating | - -**Request Body — plain values:** - -```json -{ - "title": "Updated Title" -} -``` - -**Request Body — with operators (computed updates):** - -```json -{ - "price": { "$inc": 10 }, - "stock": { "$mul": 0.5 }, - "title": "New Title" -} -``` - -**Response (200):** - -```json -{ - "modified": 5 -} -``` - -**Response (200, dry-run):** - -```json -{ - "dryRun": true, - "matched": 5 -} -``` - -**Response (200, with validation errors in per-document mode):** - -```json -{ - "modified": 3, - "failed": 2, - "errors": [ - { "id": "60a1b2c3...", "error": "validation failed", "validationErrors": {...} } - ] -} -``` - -**Execution Modes:** - -1. **Per-document (default):** Without a `bulkUpdate` hook step and without `bulk.optimize.update`, each document is updated individually. Validation runs in two phases: first a **Changeset check** (type + operator validity + `strictFields`) on the raw request body, then after operators are resolved in-memory against each document, a **full Update validation** (types + eval + required for present fields). Hooks run between the phases: `bind` → `validate` → field validation → `update` → `return`. - -2. **Optimised (with `bulkUpdate` hook or `bulk.optimize.update: true`):** The hook runs once (if defined), then all matching documents are updated in a single DB call. **Per-document hooks (`bind`, `validate`, `update`), field validation, and per-document audit are completely skipped.** This is significantly faster but bypasses all application-level validation and per-document business logic. - -> **Warning:** The optimised path skips field validation (required, type checks, `in`, `format`, `minLength`, `maxLength`, `min`, `max`, `minDate`, `maxDate`, `pattern`, `eval`, file validators) and per-document hooks. Use only when you trust the input data or handle validation in the bulk hook itself. - -> **Performance Hint:** For high-performance bulk operations (thousands of documents), enable the optimised path. This avoids N+1 queries and runs as a single database operation. When combined with operators like `$inc` or `$mul`, the update runs entirely in the database without loading any documents into memory: -> -> ```yaml -> hooks: -> put: -> bulkUpdate: -> type: javascript -> file: hooks/my_collection/bulk_update.js -> ``` -> -> The hook receives `{ filter, data }` and can modify both. After the hook, the server executes one `updateMany` call. - ---- - -### Bulk Delete Documents - -```text -DELETE /api/v1/_/:project/:collection?filter={...} -``` - -Deletes all documents matching the filter. Same permission system and pre-filters as single-document deletes. - -**Query Parameters:** - -| Parameter | Required | Description | -| --------- | -------- | ----------- | -| `filter` | Yes | JSON-encoded MongoDB filter query | -| `dryRun` | No | `1` or `true` — returns match count without deleting | - -**Response (200):** - -```json -{ - "deleted": 5 -} -``` - -**Response (200, dry-run):** - -```json -{ - "dryRun": true, - "matched": 5 -} -``` - -**Execution Modes:** - -1. **Per-document (default):** Without a `bulkDelete` hook step and without `bulk.optimize.delete`, each document is deleted individually with hooks (`delete` → [DB delete] → [file cleanup] → `return`). - -2. **Optimised (with `bulkDelete` hook or `bulk.optimize.delete: true`):** The hook runs once (if defined), then all matching documents are deleted in a single DB call. **Per-document hooks (`delete`), per-document file cleanup sequencing, and per-document audit are completely skipped.** - -> **Warning:** The optimised path skips per-document `delete` hooks. If your delete hooks contain business logic (cascading deletes, notifications), that logic will not run. - -> **Configuration without hook file:** -> -> ```yaml -> collections: -> - name: products -> bulk: -> optimize: -> delete: true -> ``` -> -> This activates the optimised path without needing a hook file. - ---- - -### POST with Method Override - -```text -POST /api/v1/_/:project/:collection -X-HTTP-Method-Override: GET -``` - -Converts a POST request to a GET request. The JSON body is converted to query parameters. This is useful for sending complex filter queries that would exceed URL length limits. - -**Example:** - -```bash -curl -X POST http://localhost:8080/api/v1/_/myblog/articles \ - -H "X-HTTP-Method-Override: GET" \ - -H "Content-Type: application/json" \ - -d '{"filter": "{\"$and\": [{\"published\": true}, {\"date\": {\"$gte\": \"2024-01-01\"}}]}", "limit": "10"}' -``` - ---- - -## LLM Endpoints - -### Chat Completion - -```text -POST /api/v1/llm/chat -``` - -**Request Body:** - -```json -{ - "provider": "openai", - "model": "gpt-4", - "messages": [ - {"role": "system", "content": "You are a helpful assistant."}, - {"role": "user", "content": "Hello!"} - ], - "stream": false, - "temperature": 0.7, - "maxTokens": 1000 -} -``` - -**Non-streaming Response (200):** - -```json -{ - "content": "Hello! How can I help you today?", - "usage": { - "promptTokens": 20, - "completionTokens": 10, - "totalTokens": 30 - } -} -``` - -**Streaming Response:** When `"stream": true`, returns `text/event-stream`: - -```text -data: {"content":"Hello"} -data: {"content":"! How"} -data: {"content":" can I help"} -data: [DONE] -``` - -**Auth:** JWT required. User must have sufficient LLM budget. - -### List Providers - -```text -GET /api/v1/llm/providers -``` - -**Response:** - -```json -[ - { - "name": "openai", - "type": "openai", - "defaultModel": "gpt-4", - "models": ["gpt-4", "gpt-3.5-turbo"] - } -] -``` - -**Auth:** None required. - ---- - -## Audit Log Endpoint - -```text -GET /api/v1/audit -``` - -Returns audit log entries. See [Audit Logging](10-audit.md) for configuration details. - -**Query Parameters:** - -| Parameter | Type | Default | Description | -| --------- | ------------- | ---------------- | ------------------------------------------------- | -| `filter` | JSON string | — | MongoDB filter query (validated against AuditLog fields) | -| `limit` | int | server default | Maximum results (capped at server config max) | -| `offset` | int | `0` | Number of results to skip | -| `sort` | string | `-insertTime` | Sort field(s), prefix `-` for descending | -| `count` | `1` or `true` | — | Include count in `X-Results-Count` header | - -**Response (200):** - -```json -[ - { - "id": "507f1f77bcf86cd799439011", - "insertTime": "2024-01-15T10:30:00Z", - "userId": "507f1f77bcf86cd799439012", - "username": "admin", - "userRole": 0, - "projectNamespace": "myblog", - "projectName": "My Blog", - "collection": "articles", - "action": "create", - "source": { - "type": "controller", - "collection": "articles", - "method": "POST" - }, - "documentId": "507f1f77bcf86cd799439013", - "snapshot": { "title": "Hello World" }, - "ip": "192.168.1.1", - "authMethod": "jwt" - } -] -``` - -**AuditLog fields:** - -| Field | Type | Description | -| ------------------ | ------ | ------------------------------------------------------------------- | -| `id` | string | Audit entry ID | -| `insertTime` | string | Timestamp of the audit event | -| `userId` | string | ID of the user who performed the action | -| `username` | string | Username of the acting user | -| `userRole` | int | Role of the acting user (0=admin, 1=user) | -| `projectNamespace` | string | Namespace of the affected project | -| `projectName` | string | Name of the affected project | -| `collection` | string | Affected collection name | -| `action` | string | Action type: `create`, `update`, `delete`, `deleteMany`, `findOneAndUpdate`, `get`, `login`, `reload`, `shutdown` | -| `source` | object | Origin of the action (see below) | -| `documentId` | string | ID of the affected document (for single-doc operations) | -| `snapshot` | object | Full document snapshot (for create) or previous state (for update/delete) | -| `changes` | object | Changed fields (for update operations) | -| `filter` | object | Filter used (for bulk operations) | -| `count` | int | Number of affected documents (for bulk operations) | -| `ip` | string | Client IP address | -| `authMethod` | string | Authentication method: `jwt` or `admin-token` | -| `tokenLabel` | string | Admin token label (when `authMethod` is `admin-token`) | -| `tokenPrefix` | string | First 8 characters of the admin token (for identification) | - -**`source` object:** - -| Field | Type | Description | -| ------------ | ------ | ------------------------------------------------------ | -| `type` | string | `controller`, `hook`, `system`, or `job` | -| `collection` | string | Source collection (when triggered from a hook) | -| `method` | string | HTTP method or hook method | -| `step` | string | Hook step (e.g. `create`, `update`) | -| `file` | string | Hook file path (when triggered from a hook) | -| `line` | int | Line number in hook file | - -**Auth:** JWT required. Admins see all entries, users see only entries for projects/collections they have GET access to. - ---- - -## Admin Endpoints - -Protected by admin tokens with `server.shutdown` permission. - -### Shutdown Server - -```text -POST /api/v1/admin/shutdown -``` - -Gracefully shuts down the server. - -**Headers:** - -```text -X-Admin-Token: your-admin-token -``` - -**Response (200):** - -```json -{ - "message": "server is shutting down" -} -``` - ---- - -### Reload Project - -```text -POST /api/v1/_/:project/_/admin/reload -``` - -Hot-reloads a project's configuration (re-reads config file, recompiles hooks, restarts jobs). - -**Auth:** Requires one of: -- Server-level admin token with `project.reload` permission (`X-Admin-Token` or `Authorization: Bearer`) -- Project-level token with `allowReload: true` -- JWT login with admin role (`X-Auth-Token`) - -**Response (200):** - -```json -{ - "message": "ok" -} -``` - ---- - -### Backup Download - -```text -GET /api/v1/_/:project/_/admin/backup -``` - -Download a full project backup as ZIP (collections + media files). Supports selective -collections via `?collections=`, and SSE progress streaming via `?sse=1`. - -**Auth:** Requires `project.backup` permission (server token), `allowBackup: true` (project token), or admin JWT. - -Detailed documentation: [Backup & Restore](21-backup-restore.md) - ---- - -### Backup Upload & Analyze - -```text -POST /api/v1/_/:project/_/admin/backup -Content-Type: multipart/form-data -``` - -Upload a backup ZIP for analysis. Returns a `restoreToken` for use in the restore phase. -Supports SSE progress streaming via `?sse=1`. No data is written in this phase. - -Detailed documentation: [Backup & Restore](21-backup-restore.md) - ---- - -### Backup Restore - -```text -POST /api/v1/_/:project/_/admin/backup/:restoreToken -``` - -Execute the restore from a previously analyzed backup. Supports `?collections=` for -selective restore, `?merge=true` for updating instead of deleting, and `?sse=1` for -progress streaming. - -Detailed documentation: [Backup & Restore](21-backup-restore.md) - ---- - -### Backup Cleanup - -```text -DELETE /api/v1/_/:project/_/admin/backup/:restoreToken -``` - -Manually clean up temp data for a restore token. Temp data is also auto-cleaned -after TTL expiry (1h for restore data, 5min for download ZIPs). - -Detailed documentation: [Backup & Restore](21-backup-restore.md) - ---- - -### Backup Download File - -```text -GET /api/v1/_/:project/_/admin/backup/download/:file -``` - -Fetch a previously generated temp backup ZIP (from SSE download mode). -One-time use — second request returns 404. - -Detailed documentation: [Backup & Restore](21-backup-restore.md) - ---- - -## Static Assets - -```text -GET /api/v1/_/:project/_/assets/:name/*path -``` - -Serves static files from a named asset directory defined in the project config. - -**Example:** - -```bash -# Serve /public/css/style.css from asset group "static" -curl http://localhost:8080/api/v1/_/myblog/_/assets/static/css/style.css -``` - ---- - -## OpenAPI Specification - -```text -GET /api/v1/_openapi/:project -``` - -Returns an auto-generated OpenAPI 3.0 specification for the project. See [OpenAPI & Metrics](13-openapi-metrics.md). - ---- - -## Error Response Format - -All error responses follow this structure: - -```json -{ - "error": "description of the error", - "errorCode": "20240115103000-a1b2c3d4" -} -``` - -The `errorCode` is a unique identifier combining a timestamp and random string, useful for log correlation. - -**With stack traces** (when `response.error.stack: true`): - -```json -{ - "error": "description of the error", - "errorCode": "20240115103000-a1b2c3d4", - "errorStack": "goroutine 1 [running]:\n...", - "callerStack": "goroutine 1 [running]:\n..." -} -``` - -**With custom error message** (when `response.error.message` is set): - -```json -{ - "error": "An internal error occurred", - "errorCode": "20240115103000-a1b2c3d4" -} -``` - -**Validation errors** additionally include: - -```json -{ - "error": "validation failed", - "errorCode": "...", - "validationErrors": { - "title": {"code": "required", "message": "data is missing"}, - "email": {"code": "format", "message": "invalid email format"}, - "name": {"code": "eval", "message": {"de": "Mindestens 3 Zeichen", "en": "Minimum 3 characters"}} - } -} -``` - -> **Note:** The `message` field can be a string or an object (e.g. for i18n), depending on the `eval` validator return value. - -## Special Response Behaviors - -- **NaN/Infinity values** in JSON responses are automatically converted to `null` -- **Gzip compression** is applied by default, except for: `.pdf`, `.mp4`, `.jpg`, `.jpeg`, `.png` extensions and the `/metrics` endpoint diff --git a/src/content/docs/server/04-collections.md b/src/content/docs/server/04-collections.md deleted file mode 100644 index 070aaed..0000000 --- a/src/content/docs/server/04-collections.md +++ /dev/null @@ -1,1749 +0,0 @@ ---- -title: 'Collections' ---- - -# Collections - -Collections are the core data model in Tibi-Server. Each collection is defined in a project's YAML config and maps to a MongoDB collection in the project's database. - -## Collection Document Model - -Every document has these automatic fields plus arbitrary dynamic data: - -```json -{ - "id": "507f1f77bcf86cd799439011", // ObjectID (hex string in JSON) - "insertTime": "2024-01-15T10:30:00Z", // Set on create - "updateTime": "2024-01-16T14:22:00Z", // Set on update - "title": "...", // Dynamic fields from data - "content": "..." -} -``` - -The `insertTime` and `updateTime` fields are managed automatically and cannot be set via the API. - -## Field Types - -| Type | Go Type | Description | Example Value | -|------|---------|-------------|---------------| -| `string` | `string` | Text | `"Hello World"` | -| `number` | `float64` | Integer or float | `42`, `3.14` | -| `float` | `float64` | Alias for number | `3.14` | -| `float64` | `float64` | Alias for number | `3.14` | -| `int` | `float64` | Integer (stored as float64 in JSON) | `42` | -| `bool` | `bool` | Boolean | `true` | -| `boolean` | `bool` | Alias for bool | `false` | -| `date` | `time.Time` | Date (string auto-parsed) | `"2024-01-15"` | -| `datetime` | `time.Time` | Date+time (string auto-parsed) | `"2024-01-15T10:30:00Z"` | -| `time` | `string` | Time string (no parsing) | `"10:30:00"` | -| `object` | `map[string]interface{}` | Nested object with subfields | `{"email": "..."}` | -| `file` | `map[string]interface{}` | File upload object | `{"src": "...", "type": "..."}` | -| `any` | `interface{}` | No type validation | Any value | - -### Array Types - -Append `[]` to any type to define an array: - -| Type | Description | Example | -|------|-------------|---------| -| `string[]` | Array of strings | `["tag1", "tag2"]` | -| `number[]` | Array of numbers | `[1, 2, 3]` | -| `object[]` | Array of objects | `[{"name": "..."}, ...]` | -| `file[]` | Array of files | (TODO — not fully implemented) | -| `{collection}[]` | Cross-reference array | `["507f...", "508a..."]` | - -### Cross-Reference Types - -Field types can reference other collection names (e.g., `employees[]`). This is purely informational for the admin UI and OpenAPI spec — no database-level foreign key enforcement. - -## Field Definition - -```yaml -fields: - - name: title # Field name (used as JSON/BSON key) - type: string # Field type - validator: # Validation rules (optional) - required: true - allowZero: false - pattern: "[A-Z]{3}-[0-9]{2}" - eval: "$this.length > 3" - index: # Field-level indexes (optional) - - single - - text - subFields: # For object/object[] types - - name: email - type: string - validator: - required: true - meta: # Arbitrary metadata (for UI, OpenAPI, etc.) - label: "Title" - description: "Article title" - openapi: - description: "The article title" -``` - -## Validation - -Validation runs during POST and PUT operations, after the `validate` hook step. - -### Validator Options - -```yaml -validator: - required: true # Field must be present and non-null - allowZero: false # When required=true: allow "" (empty string) and 0 - in: ["draft", "published"] # Value must equal one of the listed primitive values - format: email # Built-in string format validator: email, url, uuid, slug - minItems: 1 # Minimum number of entries for array fields - maxItems: 5 # Maximum number of entries for array fields - minLength: 3 # Minimum number of characters for string fields - maxLength: 20 # Maximum number of characters for string fields - min: 1 # Minimum numeric value for number/float/float64 fields - max: 10 # Maximum numeric value for number/float/float64 fields - minDate: "2024-01-01" # Earliest allowed date/time/datetime value - maxDate: "2026-12-31" # Latest allowed date/time/datetime value - pattern: "[A-Z]{3}-[0-9]{2}" # Full-string regex match, similar to HTML input pattern - minFileSize: "50KB" # Minimum decoded file size for file/file[] fields - maxFileSize: "5MB" # Max file size for file/file[] fields (supports B, KB, MB, GB, TB) - accept: ["image/png", "image/webp"] # Allowed MIME types for file/file[] fields - image: - minWidth: 800 # Minimum image width in px for newly uploaded images - maxWidth: 2400 # Maximum image width in px for newly uploaded images - minHeight: 600 # Minimum image height in px for newly uploaded images - maxHeight: 1800 # Maximum image height in px for newly uploaded images - allowNonImages: true # Optional: skip image dimension checks entirely for non-image uploads - allowUnknownDimensions: true # Optional: allow image/* uploads when dimensions cannot be determined - eval: | # JavaScript expression for custom validation - $this.length >= 3 && $this.length <= 100 -``` - -Currently implemented declarative validator options are `required`, `allowZero`, `in`, `format`, `minItems`, `maxItems`, `minLength`, `maxLength`, `min`, `max`, `minDate`, `maxDate`, `pattern`, `minFileSize`, `maxFileSize`, `accept`, `image.minWidth`, `image.maxWidth`, `image.minHeight`, `image.maxHeight`, `image.allowNonImages`, and `image.allowUnknownDimensions`. - -### Validation Behavior - -| Option | Behavior | -|--------|----------| -| `required: true` | Field must be non-null. Without `allowZero`, empty strings and `0` also fail. | -| `required: true, allowZero: true` | Field must be present, but `""` and `0` are allowed. | -| `in: [...]` | For primitive scalar fields: value must equal one of the listed allowed values. Optional string fields may still use `""`. | -| `format: email|url|uuid|slug` | For `string` fields: validates a built-in semantic format without needing a custom regex. Optional fields may still use `""`; required fields must contain a valid non-empty value. | -| `minItems: N` | For array fields: value must contain at least `N` items. | -| `maxItems: N` | For array fields: value must contain at most `N` items. | -| `minLength: N` | For `string` fields: value must contain at least `N` characters. Optional string fields may still use `""`. | -| `maxLength: N` | For `string` fields: value must contain at most `N` characters. | -| `min: N` | For `number`, `float`, and `float64` fields: value must be at least `N`. | -| `max: N` | For `number`, `float`, and `float64` fields: value must be at most `N`. | -| `minDate: "..."` | For `date`, `time`, and `datetime` fields: value must be on or after the configured bound. Bounds are parsed with the same date parser as request values. | -| `maxDate: "..."` | For `date`, `time`, and `datetime` fields: value must be on or before the configured bound. Bounds are parsed with the same date parser as request values. | -| `pattern: "..."` | For `string` fields: value must fully match the regex pattern. Optional string fields may still use `""`. | -| `minFileSize: "50KB"` | For `file` and `file[]` fields: newly uploaded files must be at least the configured decoded size. | -| `maxFileSize: "5MB"` | For `file` and `file[]` fields: max decoded file size. Exceeding returns 400. | -| `accept: ["image/png"]` | For `file` and `file[]` fields: uploaded MIME type must match one of the configured allowlist entries. | -| `image.minWidth: 800` | For newly uploaded `file` and `file[]` images: width must be at least `800px`. | -| `image.maxWidth: 2400` | For newly uploaded `file` and `file[]` images: width must be at most `2400px`. | -| `image.minHeight: 600` | For newly uploaded `file` and `file[]` images: height must be at least `600px`. | -| `image.maxHeight: 1800` | For newly uploaded `file` and `file[]` images: height must be at most `1800px`. | -| `image.allowNonImages: true` | For newly uploaded `file` and `file[]` fields with mixed media: non-image uploads bypass `validator.image`, while actual images are still checked against the configured bounds. | -| `image.allowUnknownDimensions: true` | For newly uploaded `file` and `file[]` images: if the upload is an `image/*`, but dimensions cannot be determined, the file is accepted and min/max dimension checks are skipped. | -| `eval: "..."` | JavaScript expression. Must return `true` to pass. Strings are used as error message. Objects (e.g. i18n) are passed through as-is. | - -### Validation Modes - -Validation runs in one of three modes depending on the operation: - -| Mode | Used by | Checks | Skips | -|------|---------|--------|-------| -| **Create** | POST, Bulk POST | required, allowZero, in, format, minItems, maxItems, minLength, maxLength, min, max, minDate, maxDate, pattern, eval, type, strictFields | — | -| **Update** | PUT (after operator resolve), Bulk PUT per-doc (after resolve) | in, format, minItems, maxItems, minLength, maxLength, min, max, minDate, maxDate, pattern, eval, type, strictFields | Missing/nil fields (partial update), required for absent fields | -| **Changeset** | PUT (before operator resolve), Bulk Update (initial check) | type compatibility, operator validity (e.g. `$inc` requires numeric), strictFields | required, eval | - -**Two-phase validation for PUT with operators:** - -When a PUT request contains update operators (`$inc`, `$set`, etc.), validation runs twice: - -1. **Phase 1 — Changeset:** The raw request body is validated in Changeset mode. This catches type mismatches and unknown fields before reading the existing document from the database. -2. **Phase 2 — Update:** After operators are resolved against the existing document (e.g. `$inc: 5` on `price: 10` produces `price: 15`), the concrete result is validated in Update mode with full eval and type checks. - -This two-phase approach avoids unnecessary DB reads when the changeset itself is invalid. - -### Type Validation - -Type checking is automatic based on the `type` field: - -- `string` type rejects numbers, objects, arrays -- `number`/`float`/`float64` rejects strings, objects, arrays -- `bool`/`boolean` rejects strings, numbers -- `date`, `time`, and `datetime` strings are auto-parsed via `dateparse.ParseAny()` and stored as `time.Time` -- `object` and `file` must be JSON objects -- Array types must be JSON arrays with elements matching the base type - -For date-like fields, prefer stable ISO-style input even though `dateparse.ParseAny()` accepts many common variants. - -Recommended formats: - -- `date`: `YYYY-MM-DD`, for example `2024-01-15` -- `datetime`: full ISO 8601 / RFC 3339 timestamp, for example `2024-01-15T10:30:00Z` or `2024-01-15T10:30:00+01:00` -- `time`: `HH:MM` or `HH:MM:SS`, for example `09:30` or `09:30:00` - -Accepted in practice are also many other common date strings that `dateparse.ParseAny()` understands, but these looser formats are not recommended for API clients because they can be locale- or timezone-ambiguous. - -Examples: - -- Good `date` values: `2024-01-15`, `2026-12-31` -- Good `datetime` values: `2024-01-15T10:30:00Z`, `2024-01-15T10:30:00+02:00` -- Good `time` values: `09:30`, `09:30:00` -- Avoid ambiguous values like `01/02/2024` or `15.01.2024 10:30` - -For `minDate` and `maxDate`, use the same recommended formats as the field itself. - -Comparison semantics: - -- `date`: only the calendar date is compared -- `time`: only the time-of-day is compared -- `datetime`: the full timestamp is compared - -### Eval-Based Validation - -The `eval` JavaScript expression has access to: - -| Variable | Description | -|----------|-------------| -| `$this` | Current field value | -| `$` / `entry` | Full document data | -| `$parent` | Parent object (for nested fields) | -| `$stack` | Array of parent values from root to current | -| `$project` | Project data | -| `$auth` | Current user auth data (id, username, role, permissions) | -| `$namespace` | Project namespace | -| `$method` | HTTP method (`"post"` or `"put"`) | -| `context` | Full hook context with all packages (db, http, etc.) | - -**Return values:** - -| Return | Result | -|--------|--------| -| `true` | Validation passes | -| `false` | Validation fails with message "evaluated to false" | -| `"custom message"` | Validation fails with the returned string as message | -| `{de: "...", en: "..."}` | Validation fails, object is passed through as message (i18n support) | -| `null` | Validation fails with message "evaluated to null" | -| `throw({...})` | Validation fails, exception value stored in `validationErrors[field].exception` | - -**Example:** - -```yaml -validator: - eval: | - if ($method === "post" && !$this) return "required on create"; - if ($this && $this.length < 3) return "minimum 3 characters"; - true -``` - -**i18n example (returns object as message):** - -```yaml -validator: - eval: | - if (!$this || $this.length < 3) return { - de: "Mindestens 3 Zeichen", - en: "Minimum 3 characters" - }; - true -``` - -### Validation Error Response - -```json -{ - "error": "validation failed", - "errorCode": "...", - "validationErrors": { - "title": { - "code": "required", - "message": "data is missing" - }, - "contact.email": { - "code": "eval", - "message": "invalid email format", - "exception": { "details": "..." } - } - } -} -``` - -Nested field paths use dot notation: `contact.email`, `tags.0`. - -### Strict Fields (Unknown Field Rejection) - -By default, tibi-server accepts fields that are not defined in the collection schema (lenient mode). This is useful for backward compatibility with existing projects. To reject unknown fields, set `strictFields: true`. - -**Configuration hierarchy** (most specific wins): - -| Level | Scope | Default | -|-------|-------|---------| -| Project (`config.yml`) | All collections in the project | `false` (lenient) | -| Collection | Single collection, overrides project | inherit from project | -| Object field | Sub-fields of a `type: object` field, overrides collection | inherit from collection | - -A `nil`/absent value means "inherit from parent". The effective default is `false`. - -**Project-level example:** - -```yaml -namespace: myproject -strictFields: true # all collections reject unknown fields -collections: - - name: articles - fields: - - name: title - type: string -``` - -**Collection-level override:** - -```yaml -collections: - - name: articles - strictFields: false # this collection allows unknown fields - fields: - - name: title - type: string -``` - -**Object field override:** - -```yaml -fields: - - name: details - type: object - strictFields: false # sub-fields of 'details' allow unknowns - subFields: - - name: color - type: string -``` - -**Behavior per operation:** - -- **POST / Bulk POST:** Unknown top-level fields or sub-fields of `type: object` are rejected with `unknownField` validation error. -- **PUT:** Unknown fields in the request body (or within `$set`/`$unset` etc.) are rejected. -- **Bulk Update:** Both field-level operators (`$set`, `$unset`, …) and top-level operators are checked for unknown fields. - -When `strictFields` is `false` (or absent), unknown fields pass through to MongoDB without validation. - ---- - -## CRUD Lifecycle - -### GET (Read) - -``` -1. Permission Check (MethodIsAllowed) -2. Pre-Filter from permissions (with $subst resolution) -3. Projection from config/query parameter -4. Filter query validation (whitelist check) -5. Parse ?lookup= parameter (if present) - ├── Validate source field exists in schema - ├── Auto-detect or validate target collection - ├── Permission check on target (GET must be allowed) - ├── Compute target projection (role-based) and pre-filter - └── Choose strategy: fast path ($lookup pipeline) or safe path (post-query) -6. Hook: "read" step - ├── Can modify: filter, selector, offset, limit, sort - └── Can add: pipelineMod (MongoDB aggregation pipeline modifier) -7. DB: ReadCollection (with pagination, sorting, and fast-path $lookup if applicable) -8. Audit log (if configured for "get" action) -9. Safe-path lookup resolution (if not using fast path) - ├── Collect referenced IDs from results - ├── Batch-query target collections (respecting pre-filter, projection, hooks) - └── Embed resolved documents into results -10. Hook: "return" step - └── Can override results entirely -``` - -### GET Count (/_count) - -Dedicated count endpoint: `GET /_/:project/:collection/_count` - -Returns `{"count": N}` without loading documents. Uses the same `get` permission. - -``` -1. Permission Check (MethodIsAllowed with "get") -2. Pre-Filter from permissions (with $subst resolution) -3. Filter query validation (whitelist check) -4. Hook: "read" step - └── Can modify: filter -5. DB: ReadCollectionCount -``` - -> **Note:** The `get.return` hook is **not** executed for count requests (there are no documents -> to transform). The `get.read` hook runs with `one: false` and can modify the filter. - -### POST (Create) - -``` -1. Permission Check -2. Body type detection (JSON object → single, JSON array → bulk) -3. Hook: "bind" step - └── Can provide data (skips JSON body parsing) -4. JSON body parsing (bindCollection) - └── Auto-strips insertTime, updateTime -5. Hook: "validate" step - └── Can modify data -6. Field validation (type checking, required, eval) -7. Hook: "create" step - └── Can modify data -8. File upload handling (Base64 → filesystem) -9. DB: CreateDocument -10. File rename (temp ID → real ID) -11. Audit log -12. Hook: "return" step - └── Can override results -``` - -### Bulk Create (POST with JSON array) - -When the POST body is a JSON array, the server automatically switches to -bulk create mode. This requires the `post.bulk` permission to be set. - -``` -1. Permission Check (MethodIsAllowed for "post") -2. Bulk Permission Check (BulkIsAllowed for "post" — requires post.bulk: true) -3. Parse JSON array body -4. dryRun support (?dryRun=1 returns {dryRun: true, matched: N}) -5. If "bulkCreate" hook OR bulk.optimize.create is enabled: - ──── OPTIMISED PATH (single DB call) ──── - a. Hook: "bulkCreate" step (if defined, context.data = full array) - b. Per document: validate fields, handle file uploads - c. DB: CreateDocuments (InsertMany — one call) - d. File rename (temp ID → real ID) per document - e. Audit: single "bulkCreate" entry with count - f. Hook: "bulkReturn" step (if defined) - ⚠ SKIPPED: bind, validate, create hooks per document -6. If neither hook nor bulk.optimize: - ──── PER-DOCUMENT PATH ──── - a. For each document: bind hook → validate hook → - field validation → create hook → file upload → - DB CreateDocument → file rename → audit → return hook - b. Errors are collected, successful documents continue - c. Response: {created: N, ids: [...], failed: M, errors: [...]} -``` - -### PUT (Update) - -``` -1. Permission Check -2. Changeset validation (Changeset mode: types + operators + strictFields; NO required, NO eval) -3. Read existing document from DB -4. Resolve operators against existing document ($inc, $set, etc.) -5. Hook: "bind" step - └── Can provide data (skips JSON body parsing) -6. JSON body parsing (if bind didn't provide data) -7. Hook: "validate" step -8. Field validation (Update mode: only present fields, full type + eval checks) -9. Hook: "update" step - └── Can modify data -10. File upload handling -11. Read snapshot for audit (before update) -12. DB: UpdateDocument (partial update with $set) - └── Auto-strips _id, id fields -13. Audit log (with snapshot + changes) -14. Hook: "return" step -``` - -### DELETE - -``` -1. Permission Check -2. Hook: "delete" step -3. Read snapshot for audit (before delete) -4. DB: DeleteDocument -5. Delete associated files (entire upload directory for this document) -6. Audit log (with snapshot) -7. Hook: "return" step -``` - -### Bulk Update (PUT without ID) - -``` -1. Permission Check (MethodIsAllowed for "put") -2. Bulk Permission Check (BulkIsAllowed for "put" — requires put.bulk: true) -3. Filter query validation (whitelist check) -4. JSON body parsing (update data, may contain operators like $inc, $mul, $unset) -5. Changeset validation (Changeset mode: types + operators + strictFields; NO required, NO eval) -6. If "bulkUpdate" hook OR bulk.optimize.update is enabled: - ──── OPTIMISED PATH (single DB call) ──── - a. Hook: "bulkUpdate" step (if defined, otherwise skipped) - b. DB: UpdateDocuments / UpdateDocumentsRaw (one call) - c. Audit: single "bulkUpdate" entry with filter + count - d. Hook: "bulkReturn" step (if defined) - ⚠ SKIPPED: bind, validate, full field validation, update hook, per-doc audit -7. If neither hook nor bulk.optimize: - ──── PER-DOCUMENT PATH ──── - a. Find all matching documents - b. For each document: resolve operators → bind → validate hook → - field validation (Update mode: full type + eval) → - update hook → DB update → audit - c. Hook: "return" step per document - Note: Step 5 runs Changeset validation on the raw request. - Step 7b runs full Update validation after operators are resolved. -``` - -### Bulk Delete (DELETE without ID) - -``` -1. Permission Check (MethodIsAllowed for "delete") -2. Bulk Permission Check (BulkIsAllowed for "delete" — requires delete.bulk: true) -3. Filter query validation (whitelist check) -4. If "bulkDelete" hook OR bulk.optimize.delete is enabled: - ──── OPTIMISED PATH (single DB call) ──── - a. Hook: "bulkDelete" step (if defined, otherwise skipped) - b. If collection has file fields: collect IDs for file cleanup - c. DB: DeleteDocuments (one call) - d. If file fields: clean up upload directories - e. Audit: single "bulkDelete" entry with filter + count - f. Hook: "bulkReturn" step (if defined) - ⚠ SKIPPED: per-document delete hook, per-doc audit -5. If neither hook nor bulk.optimize: - ──── PER-DOCUMENT PATH ──── - a. Find all matching documents - b. For each document: delete hook → DB delete → file cleanup → audit - c. Hook: "return" step per document -``` - -> **Config-Shortcut:** Statt einen leeren Hook zu definieren, kann `bulk.optimize` verwendet werden: -> ```yaml -> collections: -> - name: products -> bulk: -> optimize: -> create: true -> update: true -> delete: true -> ``` - ---- - -## Query Limits - -Query limits prevent unbounded result sets on collection GET (list) endpoints. - -### Configuration - -```yaml -queryLimits: - defaultLimit: 100 # Applied when no ?limit= parameter is provided - maxLimit: 500 # Upper bound — requests above this are capped -``` - -| Property | Type | Default | Description | -| -------------- | ---- | ------- | ------------------------------------------------------- | -| `defaultLimit` | int | 10000 | Limit used when the client omits the `?limit=` param | -| `maxLimit` | int | 10000 | Hard cap — requests above this return a 400 error | - -**Behavior:** - -- If `defaultLimit > maxLimit`, it is silently capped to `maxLimit`. -- If neither value is set, both default to **10,000**. -- Hooks can still override the `limit` in the `read` step (but the hook-provided value is **not** re-capped). -- Use `offset` for pagination when collections contain more documents. - -### Example - -```yaml -collections: - - name: products - queryLimits: - defaultLimit: 50 - maxLimit: 200 - fields: - - name: title - type: string -``` - -```bash -# No limit → returns 50 results (defaultLimit) -curl /api/v1/_/shop/products - -# limit=100 → returns 100 (within maxLimit) -curl /api/v1/_/shop/products?limit=100 - -# limit=999 → returns 400 error (exceeds maxLimit of 200) -curl /api/v1/_/shop/products?limit=999 -``` - ---- - -## Meta Query Params (`meta.query`) {#metaquery} - -Hooks often read extra query parameters (`ctx.request().query(name)`). Declare -them in `meta.query` so agents understand them (REST `_meta`/OpenAPI, MCP -`collection_describe` + `extra` argument). Schema: -`tibi-types/schemas/config/collection-meta.schema.json`. - -```yaml -meta: - query: - hook: { type: string, enum: [fakedata, mail], description: { de: "Test-Dispatcher" } } - view: { type: string, maxLength: 32 } - page: { type: number } -``` - -- Types: `string`, `number`, `boolean`; optional `enum`, `maxLength`, `description`. -- MCP validates `extra` against the schema (unknown/wrong-typed → `400`); - reserved keys (credential channels, core query keys) are never allowed. -- REST ignores the schema (open by design) — it is documentation. - ---- - -## Projections - -Projections control which fields are returned in query results. - -### Configuration - -```yaml -projections: - default: - select: - title: 1 - summary: 1 - publishDate: 1 - detailed: - select: - title: 1 - summary: 1 - content: 1 - publishDate: 1 - author: 1 -``` - -### Usage - -```bash -# Use named projection -GET /api/v1/_/myblog/articles?projection=detailed - -# No projection parameter → all fields returned (or default if restricted) -GET /api/v1/_/myblog/articles -``` - -### Projection Restrictions via Permissions - -Permissions can restrict which projections a user/role can use: - -```yaml -permissions: - public: - methods: - get: true - validProjections: - - default # Public can only use "default" projection - user: - methods: - get: true - validProjections: - - default - - detailed # Logged-in users can also use "detailed" -``` - -When `validProjections` is set: -- If no `projection` parameter is provided, the **first** entry in `validProjections` is used -- Only listed projections can be requested -- Admin role (0) is not restricted - ---- - -## $subst Expressions - -Permission filters can contain dynamic substitution expressions that are resolved at request time. - -### Available Variables - -```yaml -permissions: - user: - methods: - get: true - filter: - author: - $subst: - expr: "auth.id" # Resolves to the current user's ID -``` - -| Expression | Description | -|------------|-------------| -| `auth.id` | Current user's ObjectID | -| `auth.username` | Current user's username | -| `auth.role` | Current user's role (0=admin, 1=user) | -| `auth.permissions` | Array of permission strings | -| `project.*` | Project fields | -| `api.*` | API fields | - -### JavaScript Expressions - -```yaml -filter: - category: - $subst: - eval: "auth.role === 0 ? null : 'public'" -``` - -The `eval` variant executes a compiled JavaScript expression with access to the same variables. - ---- - -## Filter Operators - -The following MongoDB operators are allowed in filter queries. Unknown operators are rejected. - -### Comparison - -`$eq`, `$gt`, `$gte`, `$in`, `$lt`, `$lte`, `$ne`, `$nin` - -### Logical - -`$and`, `$or`, `$nor`, `$not` - -### Element - -`$exists`, `$type` - -### Array - -`$elemMatch`, `$size`, `$all` - -### Text - -`$text`, `$search`, `$language`, `$caseSensitive`, `$diacriticSensitive` - -### Regex - -`$regex`, `$options` - -### Geospatial - -`$geoIntersects`, `$geoWithin`, `$near`, `$nearSphere`, `$geometry`, `$maxDistance`, `$minDistance`, `$center`, `$centerSphere`, `$box`, `$polygon` - -### Aggregation (in pipeline modifiers) - -`$match`, `$group`, `$sort`, `$project`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$replaceRoot`, `$facet`, `$bucket`, `$bucketAuto`, `$sortByCount`, `$count`, `$sample`, `$redact`, `$merge`, `$out`, `$unionWith`, `$set`, `$unset` - -### Explicitly Disabled - -| Operator | Reason | -|----------|--------| -| `$where` | NoSQL injection risk | -| `$accumulator` | Server-side code execution | -| `$function` | Server-side code execution | -| `$let` | Complex scope manipulation | -| `$setField` | Field modification risk | - ---- - -## Indexes - -### Field-Level Indexes - -```yaml -fields: - - name: email - type: string - index: - - single # db.collection.createIndex({email: 1}) - - unique # db.collection.createIndex({email: 1}, {unique: true}) - - text # db.collection.createIndex({email: "text"}) - - sparse # db.collection.createIndex({email: 1}, {sparse: true}) -``` - -### Collection-Level Indexes - -```yaml -indexes: - - name: compound_index - key: - - "category" - - "-publishDate" # Descending - background: true - - name: text_search - key: - - "$text:title" # Text index - - "$text:content" - defaultLanguage: "german" -``` - -### Auto-Management - -When a project is loaded or reloaded: -1. All configured indexes are created (idempotent) -2. Indexes existing in MongoDB but **not** in the config are automatically **dropped** -3. The `_id_` index is never dropped -4. Index creation errors are logged but don't prevent project startup - -## Search (`?q=` Parameter) - -The generic `?q=` query parameter enables full-text, regex, JavaScript eval, filter template, n-gram, and vector (semantic) search on any collection endpoint (GET list + COUNT). It works out of the box without configuration via auto-fallback, and can be fine-tuned with explicit search configs. - -### API Usage - -| Parameter | Description | -|-----------|-------------| -| `q` | Search term. Triggers search filter generation. | -| `qName` | Optional. Selects a named search config. Without it, the first config is used. | -| `qExplain` | Optional. Set to `1` or `true` to include `_search.details` score breakdowns in list responses. | - -``` -GET /_/shop/products?q=laptop -GET /_/shop/products?q=ABC-123&qName=byCode -GET /_/shop/products?q=laptop&qName=semantic&qExplain=1 -GET /_/shop/products/_count?q=laptop -``` - -### Search Config - -Define named search profiles in the collection YAML: - -```yaml -search: - - name: default - mode: text # uses the collection's text index - meta: - label: Volltextsuche - icon: search - - name: byCode - mode: regex # case-insensitive $regex on specified fields - fields: [sku, barcode] - meta: - label: Artikelnummer - icon: barcode - - name: weighted - mode: regex - fields: [name, description, sku] - regex: - weights: - name: 3 # name matches count triple - sku: 2 # sku matches count double - # description defaults to weight 1 - scoring: count # count (default), binary, log - meta: - label: Gewichtete Suche - - name: smart - mode: eval # JavaScript expression returns a MongoDB filter - eval: | - (function() { - var q = context.request().query("q"); - var f = {name: {$regex: q, $options: "i"}}; - if (context.auth.role === "admin") { - return f; - } - f.public = true; - return f; - })() - meta: - label: Intelligente Suche - - name: byOwner - mode: filter # MongoDB filter template with $subst support - filter: - $and: - - owner: - $subst: - expr: "auth.id" - - name: - $regex: - $subst: - expr: "q" - $options: "i" - score: # optional: custom relevance scoring - expr: - $size: - $regexFindAll: - input: "$name" - regex: - $subst: - expr: "q" - options: "i" - meta: - label: Eigene Einträge durchsuchen -``` - -| Property | Type | Description | -|----------|------|-------------| -| `name` | string | Unique name for this search config | -| `mode` | string | `"text"`, `"regex"`, `"eval"`, `"filter"`, `"ngram"`, `"vector"`, or `"combined"` | -| `fields` | string[] | For `mode: regex`, `mode: ngram`, and `mode: vector` — fields to include | -| `eval` | string | Only for `mode: eval` — JS expression that returns a MongoDB filter | -| `filter` | object | Only for `mode: filter` — MongoDB filter template (supports `$subst`) | -| `ngram` | object | Only for `mode: ngram` — ngram-specific settings (sizes, normalize) | -| `regex` | object | Only for `mode: regex` — scoring settings (weights, scoring mode) | -| `vector` | object | Only for `mode: vector` — vector/embedding settings (provider, prefixes, overflow) | -| `rrf` | object | Rank fusion config for hybrid search (k, topK, weights) | -| `score` | object | Optional for `mode: eval` and `mode: filter` — custom scoring expression | -| `autoRegenerate` | bool | For `mode: ngram` and `mode: vector` — auto-regenerate stale docs on reload | -| `async` | bool | Explicitly control synchronous/asynchronous generation for single ops (outbox pattern) | -| `meta` | object | Free-form metadata for UI (label, icon, description, etc.) | - -**Mode details:** - -- **`text`** — Generates `{"$text": {"$search": q}}`. Requires a text index on the collection (field-level `index: [text]` or collection-level `$text:` index). Does **not** accept a `fields` property — the text index determines which fields are searched. Results are auto-sorted by `$textScore` when no explicit `sort` is provided. The relevance score is embedded in the response as `_search: {score: N, mode: "text"}` using MongoDB's `$meta: "textScore"` expression. -- **`regex`** — Generates `{"$or": [{field: {"$regex": q, "$options": "i"}}, ...]}` for each field in `fields`. Supports dot notation for nested fields (e.g. `meta.description`). Results are auto-sorted by `_search.score` descending. The response includes `_search: {score: N, mode: "regex"}`. - - **Scoring configuration** (optional `regex` block): - - `regex.weights` — Map of field name → weight (float). Default weight is 1.0. Higher weights make matches in that field contribute more to the score. - - `regex.scoring` — Scoring mode: - - `"count"` (default): Counts total regex occurrences per field via `$regexFindAll`. A field containing the pattern 3 times scores 3 × weight. - - `"binary"`: 0 or 1 per field via `$regexMatch` — only considers presence, not frequency. Scores weight × 1 if the field matches. - - `"log"`: `ln(1 + matchCount)` per field — logarithmic dampening for diminishing returns on repeated matches. Useful when frequency varies wildly between fields. -- **`eval`** — Executes a pre-compiled JavaScript expression at request time. The expression must return a MongoDB filter object (map). The generated filter is validated against the operator whitelist before use. The JS code receives the identical `context` variable that hooks get — containing all registered packages and functions: - - `context.request()` — full request object (`.query("q")`, `.query("qName")`, `.param(...)`, `.header(...)`, etc.) - - `context.api` — API config data - - `context.project` — project data - - `context.collection` — collection config data - - `context.db`, `context.http`, ... — all GlobalContext packages (same as in hooks) -- **`filter`** — A MongoDB filter template that is resolved at request time. Supports `$subst` expressions (same as permission pre-filters) to inject dynamic values. The resolved filter is validated against the operator whitelist. Available `$subst` environment variables: - - `q` — the search term - - `qName` — the selected search config name - - `auth` — auth context (`id`, `username`, `role`, `permissions`) - - `project` — project data - - `api` — API config data - **Scoring for eval/filter** (optional `score` block): Both `eval` and `filter` modes support a `score.expr` property — a raw MongoDB aggregation expression that becomes the `_search.score` value. Supports `$subst` for dynamic value injection (same syntax as `filter` templates). When `score.expr` is configured, results are auto-sorted by `_search.score` descending. When no `score` is configured, results fall back to sorting by `-updateTime` (newest first). See [Custom Scoring for Eval/Filter](#custom-scoring-for-evalfilter) below. - -- **`ngram`** — Write-time n-gram indexing for fuzzy substring search. At write time, configured fields are normalized and split into character n-grams stored in a `_search` field. At query time, the search term is similarly normalized and split, then matched via `$in` on the stored arrays. This enables fast, typo-tolerant, substring matching without `$regex` collection scans. Requires `fields` and optionally `ngram` settings. See [NGram Search Mode](#ngram-search-mode) for full details. - -- **`vector`** — Semantic (vector) search using embedding models. At write time, configured `fields` (or a custom `eval` expression) are extracted, embedded via the configured provider, and stored as dense/sparse vectors in `_search.`. At query time, `?q=` is embedded with the same provider and matched against the in-memory vector index using cosine similarity (dense) and/or an inverted index (sparse). When the provider returns both dense and sparse embeddings, scores are combined via Reciprocal Rank Fusion (RRF). Requires a `vector` block with a valid `provider` name, plus either `fields` or `eval`. See [Vector Search Mode](#vector-search-mode) for full details. - -- **`combined`** — Merges results from multiple other search configurations of the same collection using Reciprocal Rank Fusion (RRF). At query time, all referenced search configs are executed in parallel. The results and native scores of each sub-search are then combined and normalized using RRF. This mode does not write anything to the database. It requires an `rrf.weights` block where the keys are the names of the other search configs to execute. - -### Custom Scoring for Eval/Filter - -The `eval` and `filter` modes generate custom MongoDB filters where a generic score cannot be derived automatically. To enable relevance sorting, configure `score.expr` — a raw MongoDB aggregation expression that computes the score per document. - -The expression is used directly as `$addFields: {_search: {score: , mode: "eval"|"filter"}}`. Any valid MongoDB aggregation expression works — `$cond`, `$add`, `$size`, `$regexFindAll`, etc. - -**Static expression** (no dynamic values): -```yaml -search: - - name: priority_search - mode: filter - filter: - status: active - score: - expr: - $cond: - - $eq: ["$priority", "high"] - - 10 - - 1 -``` - -**Dynamic expression with `$subst`** (query term injected at request time): -```yaml -search: - - name: weighted_search - mode: filter - filter: - $or: - - name: - $regex: - $subst: {expr: 'q'} - $options: i - - description: - $regex: - $subst: {expr: 'q'} - $options: i - score: - expr: - $add: - - $multiply: - - $size: - $regexFindAll: - input: "$name" - regex: - $subst: {expr: 'q'} - options: i - - 3 - - $size: - $regexFindAll: - input: "$description" - regex: - $subst: {expr: 'q'} - options: i -``` - -The `$subst` environment for `score.expr` is identical to `filter` templates: `q`, `qName`, `auth`, `project`, `api`. - -**Fallback sort**: When no `score.expr` is configured for `eval` or `filter` modes, results are sorted by `-updateTime` (newest first) instead of relevance. This ensures a deterministic, useful default order. - -| Config | Sort | `_search` in response | -|--------|------|-----------------------| -| `score.expr` set | `-_search.score` (descending) | `{score: N, mode: "eval"\|"filter"}` | -| No `score` | `-updateTime` (newest first) | No `_search` object | - -### Auto-Fallback (No Config) - -When no `search` config is defined, `?q=` still works using this fallback chain: - -1. **Text index exists** → `$text` / `$search` (best performance, score via `$meta: "textScore"`) -2. **String fields with `single` index** → `$regex` on those fields (good performance, score = total occurrences across fields) -3. **All top-level string fields** → `$regex` on all (collection scan, slowest, score = total occurrences) - -All fallback modes auto-sort by `_search.score` descending and include `_search: {score, mode}` in the response. - -SubFields are **not** included in auto-fallback — only top-level fields. Use explicit config with dot notation to search nested fields. - -### Auto-Regeneration (`autoRegenerate`) - -For enrichment-based search modes (`ngram` and `vector`), the server can automatically detect when the search configuration has changed and regenerate stale documents in the background. - -```yaml -collections: - - name: articles - search: - - name: fuzzy - mode: ngram - fields: [title, body] - autoRegenerate: true - - name: semantic - mode: vector - fields: [title, body] - vector: - provider: nomic-v1.5 - autoRegenerate: true -``` - -**How it works:** - -1. At config load, a `configHash` is computed from all enrichment-relevant parameters (mode, fields, ngram sizes, normalize pipeline, vector provider, dimensions, document prefix, overflow/chunk settings, eval presence). -2. When a document is created or updated, the `configHash` is stored alongside the enrichment data in `_search..configHash`. -3. On project setup/reload, if `autoRegenerate: true`, the server queries for documents where `configHash` is missing or differs from the current value. -4. Stale documents are re-enriched in the background (non-blocking) using batch processing. -5. For vector configs, in-memory vector indexes are updated via `SyncVectorIndex` after each document. - -**Key characteristics:** - -- **Non-blocking** — regeneration runs in a background goroutine, project startup is not delayed. -- **Resumable** — per-document `configHash` means interrupted regeneration picks up where it left off on next restart. -- **Change-aware** — only documents with stale/missing `configHash` are processed; up-to-date documents are skipped. -- **Safe** — uses the same `EnrichSearchFields` pipeline as normal write operations. - -**When does regeneration trigger?** - -| Change | Triggers Regeneration | -|--------|----------------------| -| Adding/removing `fields` | Yes | -| Changing `ngram.sizes` | Yes | -| Changing `ngram.normalize` pipeline | Yes | -| Changing `vector.provider` | Yes | -| Changing `vector.documentPrefix` | Yes | -| Changing `vector.overflow` / chunk settings | Yes | -| Adding/removing `eval` on vector | Yes | -| Changing `rrf` weights (query-time only) | No | -| Changing `vector.queryPrefix` (query-time only) | No | -| Changing `meta` | No | - -> **Note:** Without `autoRegenerate: true`, config changes only affect newly created/updated documents. Existing documents keep their old enrichment data until manually regenerated via the `POST /_/:project/_/admin/regenerate-search` endpoint. - -### Asynchronous Enrichment (Outbox Pattern) - -By default, creating or updating a document enriched with **ngrams** happens *synchronously*. In contrast, **vector** enrichment inherently depends on slower external API connections, so it defaults to *asynchronous* (`async: true`). The synchronous processing forces the HTTP request to wait (which can be risky for vectors). You can override this via `async: true|false` on the specific search config. - -> **Note on Bulk Operations:** Vector enrichment is **always** forced to execute asynchronously during Bulk Uploads (POST) or Bulk Updates (PUT) — regardless of any `async: false` configuration — to definitively prevent HTTP timeouts mapping huge batches synchronously. Ngrams continue to strictly respect the `async` configuration during block operations. - -When configured asynchronously, tibi-server utilizes an **Outbox Pattern**: the document is saved instantly with its `hash` calculated, but deliberately omitting its `configHash` for the search configuration (it remains `null`). It is thereby flagged as "dirty" and dynamically picked up by a background worker shortly after. You can track the backlog length via the Collection Queue Status API (`GET /_/:project/:collection/_queue`). - -### Hook Integration - -The search parameters are passed to the `read` hook: - -```js -// In a read hook: -var hookData = request(); -var q = hookData.q; // search term or "" -var qName = hookData.qName; // selected search config name or "" -var filter = hookData.filter; // already includes the search filter -``` - -Hooks can inspect `q`/`qName` and further modify the `filter` if needed. - -### Filter Sanitizer - -All filters reaching MongoDB — including those generated by `eval` and `filter` search modes — are validated against a central operator whitelist (`dbcrud.ValidFilterOperators`). This blocks dangerous operators like `$where`, `$function`, `$accumulator`, and `$let` that could allow arbitrary code execution. Unknown `$`-prefixed operators are also rejected. - -### Validation Errors - -| Condition | Error | -|-----------|-------| -| `?qName=unknown` (config not found) | HTTP 400 | -| Config with unknown `mode` | Error at config load (SetupDB) + HTTP 400 at request time | -| `mode: text` without text index | Error at config load (SetupDB) | -| `mode: regex` without `fields` | Error at config load (SetupDB) | -| `mode: eval` without `eval` expression | Error at config load (SetupDB) | -| `mode: filter` without `filter` template | Error at config load (SetupDB) | -| `mode: ngram` without `fields` | Error at config load (SetupDB) | -| `mode: ngram` with invalid size (< 2 or > 10) | Error at config load (SetupDB) | -| `mode: ngram` with unknown normalizer name | Error at config load (SetupDB) | -| `mode: vector` without `vector.provider` | Error at config load (SetupDB) | -| `mode: vector` without `fields` and without `eval` | Error at config load (SetupDB) | -| `mode: vector` with unknown provider name | Error at config load (SetupDB) | -| `eval`/`filter` produces blocked operator | HTTP 400 | - -### NGram Search Mode - -The `ngram` mode enables fuzzy substring search by pre-computing character n-grams at write time. This is ideal for searching chemical names, product codes, or any data where users type partial strings. - -#### How It Works - -1. **Write time**: When a document is created or updated, the configured fields are extracted, normalized (lowercased, special chars removed, optional transliterations), and split into character n-grams of the configured sizes. The resulting n-gram arrays are stored in a `_search..n` field on the document. -2. **Query time**: The search term `?q=` is normalized with the same pipeline, split into n-grams, and matched via `$in` against the stored arrays. Multiple search terms (space-separated) are combined with `$and` (all terms must match). For longer terms, the engine can additionally require a minimum overlap count via `ngram.queryFilter.minMatchRatio` to reduce broad candidate sets before ranking. -3. **Relevance ranking**: A relevance score is computed per document using a MongoDB `$addFields` aggregation stage. The pipeline overwrites the internal `_search` field with an API-facing object `{score: N, mode: "ngram"}`. The score counts how many of the query's n-grams match the document's stored n-grams (via `$setIntersection` + `$size`). For multi-term queries the per-term scores are summed. Results are auto-sorted by `_search.score` descending (most relevant first). An explicit `?sort=` parameter overrides this auto-sort. The `_search` object **is included** in API responses so clients can use it for display or further ranking. Fused modes such as `vector` and `combined` additionally populate `_search.details` with a per-source score breakdown. The generic structure is designed for reuse by future search modes (vector, combined, reranking). -4. **Storage**: The internal `_search` field (n-gram arrays) is automatically indexed and only exposed as the `{score, mode}` metadata object when a search pipeline is active. Without a search query, `_search` is stripped from responses. Users cannot set `_search` via POST/PUT — it is always auto-generated. - -#### Config - -```yaml -search: - - name: fuzzy - mode: ngram - fields: - - name - - synonyms # string[] fields are supported - - meta.description # dot notation for nested fields - ngram: - sizes: [2, 3] # optional, default: [2, 3] - normalize: - use: [german-umlauts, greek] # optional: restrict normalizers - eval: | # optional: append custom JS normalizer - function(text) { - return text.replace(/[^\w\s]/g, ''); - } -``` - -#### NGram Settings (`ngram:`) - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `sizes` | int[] | `[2, 3]` | N-gram sizes to generate (2 = bigrams, 3 = trigrams). Range: 2–10 | -| `normalize` | object | — | Optional normalizer configuration | -| `queryFilter` | object | — | Optional query-time prefilter tuning for large candidate sets | -| `queryFilter.minMatchRatio` | float | `0.4` | Minimum fraction of query n-grams that must overlap for the strict prefilter. Set `0` to disable it. Range: 0–1 | -| `normalize.use` | string[] | all | Restrict to named normalizers (see below) | -| `normalize.eval` | string | — | JS function appended to the normalizer pipeline. Receives the normalized string, must return a string. | - -#### Normalizer Pipeline - -The normalizer pipeline transforms text before n-gram generation. It always runs in this order: - -1. **Additional normalizers** (selectable via `normalize.use`): - - `greek` — Transliterates Greek letters to Latin equivalents (α→a, β→b, γ→g, δ→d, ε→e, ζ→z, η→e, θ→th, ι→i, κ→k, λ→l, μ→m, ν→n, ξ→x, ο→o, π→p, ρ→r, σ/ς→s, τ→t, υ→y, φ→ph, χ→ch, ψ→ps, ω→o) - - `hyphen-remove` — Removes hyphens (`-`) for compound word matching - - `german-umlauts` — Converts umlauts (ä→ae, ö→oe, ü→ue, ß→ss, including uppercase Ä/Ö/Ü) - -2. **Default normalizer** (always runs): - - Lowercase - - NFKD Unicode decomposition (strips accents: é→e, ñ→n, etc.) - - Remove all non-alphanumeric characters except spaces - - Normalize whitespace (collapse multiple spaces) - -3. **Optional eval** (runs last, if configured): - - Custom JS function receives the normalized string as its argument and must return the processed string: - ```yaml - eval: "function(text) { return text.replace(/x/g, 'ks') }" - ``` - - Arrow functions work too: `eval: "(text) => text.replace(/x/g, 'ks')"` - -**Without `normalize.use`**: ALL additional normalizers (greek, hyphen-remove, german-umlauts) run automatically. Specify `normalize.use` to restrict to a subset. - -**Combining**: `normalize.use` and `normalize.eval` can be combined. The chain runs `use` normalizers → default → eval. - -#### Examples - -**Chemical substances with Greek letters:** -```yaml -search: - - name: fuzzy - mode: ngram - fields: [name, synonyms] - ngram: - sizes: [2, 3] - # All normalizers active by default → α-Tocopherol → "a tocopherol" → ngrams -``` - -Query: `?q=tocoph&qName=fuzzy` → finds "α-Tocopherol" -Query: `?q=vitamin tocop&qName=fuzzy` → finds docs matching both terms (AND), ranked by relevance (highest `_search.score` first) - -**German product names:** -```yaml -search: - - name: suche - mode: ngram - fields: [title, description] - ngram: - normalize: - use: [german-umlauts] # only german umlauts, not greek -``` - -Query: `?q=oelsaeure&qName=suche` → finds "Ölsäure" - -**Minimal config (defaults):** -```yaml -search: - - name: search - mode: ngram - fields: [name] - # ngram.sizes defaults to [2, 3], all normalizers active -``` - -### Vector Search Mode - -The `vector` mode enables semantic (meaning-based) search using embedding models. Unlike keyword-based modes, vector search understands synonyms, paraphrases, and conceptual similarity. - -#### How It Works - -1. **Write time**: When a document is created or updated, the configured `fields` are extracted (or a custom `eval` JS expression transforms the document into a single text string). The text is then embedded via the configured provider (e.g. BGE-M3, multilingual-e5) to produce dense and/or sparse vectors. These are stored in `_search.` as BSON binary (dense: float32 LE) and map (sparse: `map[uint32]float32`). A SHA-256 input hash prevents redundant re-embedding when source fields haven't changed. -2. **Index loading**: At project startup, vectors are loaded into an in-memory index (per collection per search config). Loading happens asynchronously in the background — search returns partial results during loading. -3. **Query time**: The search term `?q=` is embedded with the same provider (using `queryPrefix` if configured). The resulting vector is matched against the in-memory index: - - **Dense**: Brute-force cosine similarity (O(n)) - - **Sparse**: Inverted index lookup with dot-product scoring - - **Hybrid**: When the provider returns both dense and sparse embeddings, scores are combined via Reciprocal Rank Fusion (RRF) - -#### Config - -```yaml -search: - - name: semantic - mode: vector - fields: - - title - - description - - content - vector: - provider: bge-m3 # referenced by name; the provider itself is configured system-wide in config.yml - documentPrefix: "search_document: " # optional, prepended before embedding - queryPrefix: "search_query: " # optional, prepended before query embedding - overflow: truncate # "truncate" (default) | "chunk" - rrf: # optional rank fusion tuning - k: 60 # RRF constant (default: 60) - topK: 50 # max results (default: 50) - weights: # per-source weights - dense: 0.7 - sparse: 0.3 -``` - -#### Vector Settings (`vector:`) - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `provider` | string | `""` | Name of the embedding provider to use. If omitted, it automatically uses the first/only provider configured system-wide. Providers are defined in `config.yml` under `server.embedding.providers` or via ENV. | -| `documentPrefix` | string | `""` | Prefix prepended to text before generating document embeddings (model-specific, e.g. `"search_document: "` for E5 models) | -| `queryPrefix` | string | `""` | Prefix prepended to the search term before generating query embeddings (e.g. `"search_query: "`) | -| `overflow` | string | `"truncate"` | How to handle text exceeding the model's max token limit: `"truncate"` (cut off) or `"chunk"` (split into overlapping chunks) | -| `chunkSize` | int | `512` | Token count per chunk when `overflow: chunk` | -| `chunkOverlap` | int | `50` | Token overlap between adjacent chunks when `overflow: chunk` | - -#### Rank Fusion Settings (`rrf:`) - -When the embedding provider returns both dense and sparse vectors, scores are merged via Reciprocal Rank Fusion. The `rrf` block tunes this behavior. The same block is also used by `combined` mode to merge sub-search results. - -| Property | Type | Default | Description | -|----------|------|---------|-------------| -| `k` | int | `60` | RRF constant — higher values give more weight to lower-ranked documents | -| `topK` | int | `50` | Maximum number of results returned from rank fusion | -| `weights` | map | `{}` | Per-source weights (e.g. `dense: 0.7, sparse: 0.3`). Unweighted sources default to 1.0 | -| `strategy` | string | `"rrf"` | Fusion strategy: `"rrf"` (pure rank-based), `"score_rrf"` (score × RRF), `"score_sum"` (direct score sum), `"rank_score"` (rank + score additive), or `"hybrid"` (configurable blend) | -| `normalize` | string | `"none"` | Score normalization across sources: `"none"`, `"minmax"` (scale to `[0,1]`), or `"zscore"` (z-score standardization). Required for score-aware strategies when source score ranges differ | -| `scoreWeight` | float | `0.5` | Blend factor for `"hybrid"` strategy: `0.0` = pure rank (RRF), `1.0` = pure score. Ignored for other strategies | -| `sourceConfig` | map | `{}` | Per-source overrides (see below) | - -**Available strategies:** - -| Strategy | Formula | Use case | -|----------|---------|----------| -| `"rrf"` | `weight × Boost / (k + rank)` | Pure rank fusion — ignores source scores. Good when score scales are incomparable | -| `"score_rrf"` | `weight × Boost × score / (k + rank)` | RRF weighted by source score. High-scoring results keep their advantage | -| `"score_sum"` | `weight × Boost × score` | Direct weighted score sum. Requires comparable scores across sources (use `normalize: minmax`) | -| `"rank_score"` | `weight × Boost × (1/(k+rank) + score)` | Additive rank + score. Both rank position and score magnitude matter | -| `"hybrid"` | `weight × Boost × (α×score + (1-α)/(k+rank))` | Configurable blend — α controlled by `scoreWeight` | - -**Per-source overrides (`sourceConfig`):** - -Each source in `weights` can have fine-grained overrides: - -| Property | Type | Description | -|----------|------|-------------| -| `scoreWeight` | float | Override the global `scoreWeight` for this specific source | -| `boost` | float | Score multiplier for this source (default `1.0`). `boost: 2.0` doubles this source's score contribution | - -Example — vector search with score-weighted dense+sparse fusion: - -```yaml -search: - - name: semantic - mode: vector - fields: [title, description] - vector: - provider: bge-m3 - rrf: - strategy: score_rrf - normalize: minmax - weights: - dense: 0.7 - sparse: 0.3 -``` - -Example — hybrid combined search with per-source tuning: - -```yaml -search: - - name: smart - mode: combined - rrf: - strategy: hybrid - normalize: minmax - scoreWeight: 0.3 - weights: - semantic: 0.8 - fuzzy: 0.2 - sourceConfig: - semantic: - scoreWeight: 0.7 - boost: 1.2 - fuzzy: - scoreWeight: 0.1 -``` - -> **Note:** When `strategy` or `normalize` are not set, the system behaves exactly as before (pure RRF, no score normalization). All new features are opt-in. - -#### Response Score Breakdown - -Vector search responses always include `_search.score` and `_search.mode`. When `qExplain=1` or `qExplain=true` is set on the list request and the query uses both dense and sparse vectors, the response also includes `_search.details` with the exact dense/sparse scores and their RRF contributions. - -- `_search.score`: final score used for sorting this result at the current level -- `_search.details.algorithm`: always `"rrf"` -- `_search.details.strategy`: the fusion strategy used (e.g. `"score_rrf"`, `"hybrid"`). Omitted when default (`"rrf"`) -- `_search.details.normalize`: the normalization mode (e.g. `"minmax"`). Omitted when `"none"` -- `_search.details.sources..score`: raw score produced by that source before fusion -- `_search.details.sources..normalizedScore`: normalized score (only present when `normalize` is active) -- `_search.details.sources..contribution`: contribution of that source to the current level's final score -- `_search.details.sources..details`: nested breakdown for sub-fusions - -Dense scores are cosine similarities. Sparse scores are sparse dot-product scores. When normalization is active, the raw score is always preserved in `.score` and the normalized value in `.normalizedScore`. - -Example response fragment for a score-rrf hybrid vector result: - -```json -{ - "_search": { - "mode": "vector", - "score": 0.016162, - "details": { - "algorithm": "rrf", - "strategy": "score_rrf", - "normalize": "minmax", - "k": 60, - "sources": { - "dense": { - "mode": "dense", - "matched": true, - "rank": 1, - "weight": 0.7, - "score": 0.84271, - "normalizedScore": 0.95, - "contribution": 0.011475 - }, - "sparse": { - "mode": "sparse", - "matched": true, - "rank": 4, - "weight": 0.3, - "score": 18.227, - "contribution": 0.004687 - } - } - } - } -} -``` - -#### Using `eval` Instead of `fields` - -For complex document structures or when you want full control over what text gets embedded, use `eval` instead of `fields`. The JS expression must be a **function** that receives the full document and returns a single string: - -```yaml -search: - - name: semantic - mode: vector - eval: | - function(doc) { - var parts = [doc.title]; - if (doc.tags) parts.push(doc.tags.join(", ")); - if (doc.meta && doc.meta.description) parts.push(doc.meta.description); - return parts.join(" — "); - } - vector: - provider: bge-m3 - queryPrefix: "search_query: " -``` - -Arrow functions work too: - -```yaml -eval: "(doc) => doc.title + ' ' + doc.description" -``` - -When only `eval` is set (without `fields`), the enrichment selector loads the full document (no field projection) since the eval expression may access any field. - -You can combine `fields` and `eval` to get an optimized MongoDB projection while still using a custom eval function. In that case, only the listed fields are loaded from the database, and `doc` in the eval function will only contain those fields: - -```yaml -search: - - name: semantic - mode: vector - fields: [title, tags, meta] - eval: | - function(doc) { - var parts = [doc.title]; - if (doc.tags) parts.push(doc.tags.join(", ")); - if (doc.meta && doc.meta.description) parts.push(doc.meta.description); - return parts.join(" — "); - } - vector: - provider: bge-m3 -``` - -> **Note:** When `fields` is set alongside `eval`, the `doc` argument only contains the projected fields — not the full document. Make sure all fields accessed in `eval` are listed in `fields`. - -#### Examples - -**Simple semantic search on product titles:** -```yaml -search: - - name: semantic - mode: vector - fields: [name, description] - vector: - provider: bge-m3 -``` - -Query: `?q=organic face cream&qName=semantic` → finds products about natural skincare, even without exact keyword matches. - -**Multilingual search with E5 model:** -```yaml -search: - - name: search - mode: vector - fields: [title, content] - vector: - provider: e5-multilingual - documentPrefix: "passage: " - queryPrefix: "query: " -``` - -Query: `?q=Hautpflege&qName=search` → finds documents about "skincare" in English content (cross-lingual). - -**Combined with ngram for fuzzy + semantic:** -```yaml -search: - - name: fuzzy - mode: ngram - fields: [name, synonyms] - ngram: - sizes: [2, 3] - - name: semantic - mode: vector - fields: [name, description] - vector: - provider: bge-m3 -``` - -Use `?q=tocoph&qName=fuzzy` for partial string matches, or `?q=vitamin E antioxidant&qName=semantic` for meaning-based search. - -#### Caching - -Vector search uses a two-level in-memory cache to avoid redundant work: - -1. **Embedding cache**: The result of `provider.Embed(query)` is cached per query text. Repeated searches with the same `?q=` (e.g. paginating through results) skip the embedding provider entirely. -2. **Scored-results cache**: The scored+filtered document list (index search → filter intersection) is cached per `collection + config + generation + query + filter`. Pagination slices the cached list; `/_count` reuses the same result. - -Cache invalidation is automatic: a **generation counter** on the vector index increments on every document Add/Remove. Cache keys include the generation, so stale entries become unreachable after index changes. LRU eviction (max 1000 entries) and a 5-minute TTL provide safety bounds. - -#### Async Vector Enrichment - -Search enrichment during write operations (POST, PUT, bulk) is split by mode: - -- **Ngram enrichment** runs **synchronously** before the document write, except when `async: true` is set, in which case it is queued. -- **Vector enrichment** defaults to `async: true` and runs in a central **background search worker queue**. For mass processing modes like Bulk Update/Post, vector operations are *always* forced into the asynchronous queue — overriding any potential `async: false` configuration — to definitively avoid HTTP timeouts in front of external LLM APIs. - -The central queue uses an Outbox pattern: -1. The write operation (POST/PUT/bulk) commits the document. For fields queued for asynchronous enrichment, the `configHash` in `_search` is set to null. -2. The write operation triggers the background worker queue (or the batch worker picks it up on rotation). -3. The queue worker queries the database for pending documents (where `configHash` is null or does not match the active config). -4. The worker processes documents in batches, sending large grouped requests (grouped by provider and prefix) to the embedding providers. -5. The worker writes `_search` data back to the document (setting the correct `configHash`) and syncs the in-memory vector index. - -**Crash recovery:** If the server crashes, `autoRegenerate` detects documents with missing or stale `configHash` natively and re-enriches them. This outbox pattern applies to all write paths. - -### Combined Search Mode - -The `combined` search mode allows you to execute multiple other search configurations in parallel and merge their results using Reciprocal Rank Fusion (RRF). This is highly useful when you want to combine different search strategies—like an exact text search, a fuzzy ngram search, and a semantic vector search—into a single ranked result list. - -Unlike `vector` or `ngram` modes, `combined` does not create its own `_search` data in the database. Instead, it delegates the actual search execution to the configurations defined in its `rrf.weights` map. - -#### Example - -```yaml -search: - - name: title_exact - mode: text - - name: fuzzy - mode: ngram - fields: [title, description] - ngram: - sizes: [3] - - name: semantic - mode: vector - fields: [description] - vector: - provider: bge-m3 - - name: my_super_search - mode: combined - rrf: - k: 60 # RRF tuning parameter (default 60) - topK: 100 # Pre-fetch limit per sub-configuration (default 100) - weights: - title_exact: 2.0 # Boost exact matches - semantic: 1.5 # Semantic meaning has high priority - fuzzy: 1.0 # Fallback fuzzy matches -``` - -Querying: `?q=my search term&qName=my_super_search` - -At query time, the system will: -1. Run `title_exact` using `mode: text` -2. Run `fuzzy` using `mode: ngram` -3. Run `semantic` using `mode: vector` -4. Gather up to `topK` (100) results from each -5. Apply the RRF formula to the ranks: `score = weight * (1 / (k + rank))` -6. Sum the RRF scores for documents found by multiple sources -7. Return the final sorted list of documents - -#### Combined Response Score Breakdown - -When `qExplain=1` or `qExplain=true` is set on the list request, combined search also returns `_search.details`. The sources are the referenced search configs from `rrf.weights`. Each source entry exposes its own local score, rank, weight, and contribution to the combined final score. - -If one of those sources is itself a hybrid vector search, its entry contains a nested `details` block with the vector-internal dense/sparse breakdown. - -Example response fragment: - -```json -{ - "_search": { - "mode": "combined", - "score": 0.021001, - "details": { - "algorithm": "rrf", - "k": 60, - "sources": { - "semantic": { - "mode": "vector", - "matched": true, - "rank": 1, - "weight": 0.7, - "score": 0.016162, - "contribution": 0.011475, - "details": { - "algorithm": "rrf", - "k": 60, - "sources": { - "dense": { - "mode": "dense", - "matched": true, - "rank": 1, - "weight": 0.7, - "score": 0.84271, - "contribution": 0.011475 - }, - "sparse": { - "mode": "sparse", - "matched": true, - "rank": 4, - "weight": 0.3, - "score": 18.227, - "contribution": 0.004687 - } - } - } - }, - "fuzzy": { - "mode": "ngram", - "matched": true, - "rank": 2, - "weight": 0.3, - "score": 0.78125, - "contribution": 0.004839 - } - } - } - } -} -``` - -#### Search Debug Headers - -When `X-Search-Debug: true` is set on the request header, the response includes three additional HTTP headers with timing and source statistics. This is useful for tuning `strategy`, `normalize`, and per-source `boost` settings. - -| Header | Example | Description | -|--------|---------|-------------| -| `X-Search-Config` | `strategy=hybrid;normalize=minmax;scoreWeight=0.30` | Fusion strategy and normalization in use | -| `X-Search-Timing` | `semantic=12ms;fuzzy=8ms;rrf=1ms;total=22ms` | Per-sub-search execution time and total | -| `X-Search-Sources` | `semantic(totalDocs=150,returnedDocs=50,scoreRange=0.10-0.95);fuzzy(...)` | Per-source statistics: candidate count, returned count, and score range after normalization | - -> **Note:** Debug headers are only emitted for search list responses (not count requests). The data is also used internally by the per-document `_search.details` breakdown. - ---- - -#### Admin Regenerate Endpoint - -Search enrichment data (`_search`) is automatically kept in sync on every POST/PUT. A SHA-256 input hash is stored alongside the enrichment data (e.g. `_search.fuzzy.hash`) so that unchanged documents are skipped during writes — this avoids redundant ngram generation or embedding API calls. - -If you change the search config (add/remove fields, change sizes/normalizers), existing documents won't have matching `_search` data. Use the admin endpoint to regenerate: - -``` -POST /_/:project/_/admin/regenerate-search -``` - -**Auth:** Requires one of: -- Server-level admin token with `project.regenerate-search` permission (`X-Admin-Token` or `Authorization: Bearer`) -- Project-level token with `allowRegenerateSearch: true` -- JWT login with admin role (`X-Auth-Token`) - -**Request body (JSON):** - -| Property | Type | Description | -|----------|------|-------------| -| `collection` | string | Optional. Limit to a specific collection | -| `modes` | string[] | Optional. Only process these search modes (e.g. `["ngram"]`) | -| `names` | string[] | Optional. Only process these named search configs | -| `ids` | string[] | Optional. Only regenerate for specific document IDs | -| `filter` | string | Optional. JSON MongoDB filter to select documents | -| `force` | bool | Optional. Ignore stored input hashes and regenerate all enrichment data. Use after config changes (different normalizer, fields, sizes) where the input text hasn't changed but the processing pipeline has. Default: `false` | - -**Examples:** - -```bash -# Regenerate all ngram search data for the "substances" collection -curl -X POST "/_/shop/_/admin/regenerate-search" \ - -H "X-Admin-Token: your-token" \ - -d '{"collection": "substances", "modes": ["ngram"]}' - -# Regenerate vector embeddings for a collection -curl -X POST "/_/shop/_/admin/regenerate-search" \ - -H "X-Admin-Token: your-token" \ - -d '{"collection": "products", "modes": ["vector"]}' - -# Regenerate a specific named search config -curl -X POST "/_/shop/_/admin/regenerate-search" \ - -H "X-Admin-Token: your-token" \ - -d '{"collection": "products", "names": ["semantic"]}' - -# Regenerate specific documents -curl -X POST "/_/shop/_/admin/regenerate-search" \ - -H "X-Admin-Token: your-token" \ - -d '{"collection": "substances", "ids": ["507f1f77bcf86cd799439011"]}' - -# Regenerate with filter -curl -X POST "/_/shop/_/admin/regenerate-search" \ - -H "X-Admin-Token: your-token" \ - -d '{"collection": "substances", "filter": "{\"category\": \"vitamin\"}"}' - -# Force regeneration after config change (ignores input hashes) -curl -X POST "/_/shop/_/admin/regenerate-search" \ - -H "X-Admin-Token: your-token" \ - -d '{"collection": "substances", "force": true}' -``` - -**Response:** -```json -{ - "results": { - "substances": { - "processed": 150, - "updated": 148, - "skipped": 2, - "errors": 0, - "errorMessages": [] - } - } -} -``` - -- **processed**: Total documents examined -- **updated**: Documents where enrichment data was (re)generated -- **skipped**: Documents where the input hash matched (no change needed). Always 0 when `force: true`. -- **errors**: Documents that failed during enrichment - -#### Memory Usage & Performance - -Vector search keeps all vectors in RAM for fast query-time access. Estimate memory per collection: - -``` -RAM ≈ numDocs × dimensions × 4 bytes (dense, float32) -``` - -| Documents | Dimensions | Dense RAM | -|-----------|------------|---------------| -| 10,000 | 384 | ~15 MB | -| 10,000 | 1024 | ~40 MB | -| 100,000 | 384 | ~150 MB | -| 100,000 | 1024 | ~400 MB | -| 1,000,000 | 1024 | ~4 GB | - -Sparse vectors use additional memory proportional to the number of non-zero entries per document (typically 50–200 entries × 8 bytes each). - -With `overflow: chunk`, each document produces multiple vectors (one per chunk). A document chunked into 5 pieces uses 5× the dense RAM. Plan accordingly for long-text collections. - -**Benchmark results** (Go benchmarks, AMD EPYC 9634, 16 cores): - -| Operation | Corpus | Latency | Allocs | -|-----------|--------|---------|--------| -| **Dense search** Top-10 | 10K × 384d | ~7 ms | 4 | -| **Dense search** Top-10 | 100K × 1024d | ~16 ms | 4 | -| **Sparse search** Top-10 | 10K × 1024d | ~0.12 ms | 19 | -| **Sparse search** Top-10 | 100K × 1024d | ~1.5 ms | 50 | -| **Hybrid search** Top-50 | 10K × 1024d | ~16 ms | 106 | -| **Hybrid search** Top-50 | 100K × 1024d | ~200 ms | 588 | -| **RRF merge** 2 sources | 10K results | ~3 ms | 83 | -| **Chunk dedup** | 100 chunks / 20 docs | ~5 µs | 10 | -| **Concurrent dense** (16 threads) | 10K × 1024d | ~0.8 ms/op | 4 | -| **Concurrent hybrid** (16 threads) | 10K × 1024d | ~1.2 ms/op | 106 | -| **Index build** dense | 10K × 384d | ~12 ms | 10K | -| **Index build** dense | 100K × 1024d | ~4.4 s | 102K | -| **Index build** hybrid | 100K × 1024d | ~78 s | 1.3M | - -**Performance characteristics:** -- **Dense search**: Brute-force cosine similarity, O(n) per query. Scales linearly with corpus size. -- **Sparse search**: Inverted index, sub-linear. Typically 10–100× faster than dense for the same corpus size. -- **Hybrid**: Both searches run, then RRF merge. Slightly more expensive than either alone, but significantly more accurate. -- **Concurrency**: Read-heavy workloads scale well across cores (RWMutex). Write operations (Add/Remove) briefly block readers. -- **Index loading**: Background async at startup. Large collections (100k+) may take a few seconds to build. -- **Embedding latency**: Native ONNX models: ~1–10 ms per text. External API providers: depends on network and provider. - -#### Supported Embedding Providers - -| Type | Description | Build Requirement | -|------|-------------|-------------------| -| `native` | Local ONNX model (e.g. BGE-M3, multilingual-e5-small). Runs in-process, no network calls. | Requires `cgo_embedding` build tag and ONNX Runtime C library | -| `openai` | OpenAI Embeddings API | None | -| `openai-compatible` | Any OpenAI-compatible API (e.g. vLLM, LiteLLM) | None | -| `ollama` | Ollama local models | None | - -Native providers require the ONNX model directory to contain `model.onnx` (and optionally `sparse_model.onnx` for sparse/hybrid support, `tokenizer.json`). - -#### Chunking (overflow: chunk) - -When a document's text exceeds the model's token context window, the `overflow` setting controls the behavior: - -- **`truncate`** (default): Text is cut off at the model's max token limit. Simple, predictable, lower RAM. Best for short-to-medium texts. -- **`chunk`**: Text is split into overlapping chunks using sentence-aware boundaries. Each chunk is embedded separately and stored as multiple vectors. At query time, all chunk vectors are searched, then deduplicated per document (best-matching chunk wins). - -Chunk settings: -- `chunkSize` (default: 512): Maximum tokens per chunk -- `chunkOverlap` (default: 50): Overlapping tokens between adjacent chunks for context continuity - -Use chunking for long-form content (articles, legal documents, transcripts) where relevant passages may appear anywhere in the text. diff --git a/src/content/docs/server/05-authentication.md b/src/content/docs/server/05-authentication.md deleted file mode 100644 index 6896fd2..0000000 --- a/src/content/docs/server/05-authentication.md +++ /dev/null @@ -1,606 +0,0 @@ ---- -title: 'Authentication & Authorization' ---- - -# Authentication & Authorization - -## Overview - -Tibi-Server uses JWT-based authentication with a role and permission system. Access control is configured per collection via permission sets that support public access, role-based access, token-based access, and custom permissions. - -## Login Flow - -### Username/Password Login - -``` -POST /api/v1/login -Content-Type: application/json - -{"username": "admin", "password": "admin"} -``` - -1. Password is hashed with MD5 and compared against the stored hash -2. If matched, a JWT token and refresh token are generated -3. JWT token is returned in the response body -4. Refresh token is set as an HttpOnly cookie - -### Refresh Token Login - -``` -POST /api/v1/login -Cookie: tibi-refresh-token=eyJ... -``` - -When no username/password is provided but a refresh token cookie exists: -1. The refresh token JWT is validated (HMAC-SHA256, expiration required) -2. User ID is extracted from the token claims -3. User is looked up by ID -4. New JWT token is issued - -### Logout - -``` -POST /api/v1/login?logout=1 -``` - -Clears the refresh token cookie (sets expiry to -1). Any non-empty `logout` query parameter value triggers logout. - -### Impersonation (Admin/OrgAdmin → User) - -``` -POST /api/v1/login/impersonate -X-Auth-Token: -Content-Type: application/json - -{"userId": ""} -``` - -Allows a SystemAdmin (role=0) or OrgAdmin to obtain a JWT token acting as another user. -This is useful for debugging permission issues ("why can User X not see document Y?"). - -**Security rules:** -| Role | Can impersonate | Reason | -|---|---|---| -| SystemAdmin (0) | Any user | Admin already has full access | -| OrgAdmin | Users in same org(s) | OrgAdmin can already grant themselves the same team permissions | -| User (1) | Nobody | Would break the permission model | - -**Response (200):** -```json -{ - "token": "", - "user": { "id": "...", "username": "targetuser", "role": 1, ... }, - "impersonatedBy": { - "id": "", - "username": "admin", - "role": 0 - } -} -``` - -**Important:** The JWT token contains an `impersonatedBy` field in the `object` claim, -which is used by the audit system to record who actually performed each action. -Every audit entry during an impersonated session will include: -- `userId`/`username`/`userRole` → the impersonated (target) user -- `impersonatedBy` → the original admin who initiated the impersonation -- `authMethod` → `"impersonation"` - -**Logout from impersonation:** Same as normal logout (`POST /api/v1/login?logout=1`). -The refresh token cookie is cleared, ending the impersonated session. The admin -must re-authenticate with their own credentials. - -**Chained impersonation:** If an admin impersonates an org-admin, and that -org-admin then impersonates a user, the `impersonatedBy` field always points to -the **original admin** who started the chain — not the intermediate impersonator. - -**Errors:** -| Status | Description | -|---|---| -| 400 | `userId is required` or invalid userId format | -| 403 | `impersonation not allowed: insufficient role` | -| 403 | `impersonation not allowed: not admin of target user's org` | -| 404 | target user not found | - -## JWT Token Structure - -### Access Token - -| Field | Value | Description | -|-------|-------|-------------| -| Algorithm | HS256 | HMAC-SHA256 | -| `object` | User data | `{id, username, role, permissions, orgs, teams, primaryOrgId, meta, ...}` | -| `type` | `"User"` | Token type identifier | -| `exp` | now + 60min | Expiration timestamp | - -The `password` field is explicitly excluded from the token (via custom JSON marshaling). - -The `object` includes org and team membership if the user has any: - -```json -{ - "object": { - "id": "60a1b2c3d4...", - "username": "alice", - "role": 1, - "permissions": [], - "orgs": [{"orgId": "...", "role": "admin"}], - "teams": [{"teamId": "...", "role": "member"}], - "primaryOrgId": "..." - }, - "type": "User", - "exp": 1700000000 -} -``` - -See [Orgs & Teams](18-orgs-teams.md) for details on org/team-based access control. - -**Auth Header:** - -``` -X-Auth-Token: eyJhbGciOiJIUzI1NiIs... -``` - -### Refresh Token - -| Field | Value | Description | -|-------|-------|-------------| -| Algorithm | HS256 | HMAC-SHA256 | -| `object` | `{id: "userId"}` | Minimal user reference | -| `type` | `"RefreshTokenData"` | Token type identifier | -| `exp` | now + 10h | Expiration timestamp | - -**Cookie:** - -| Property | Value | -|----------|-------| -| Name | `tibi-refresh-token` | -| MaxAge | 36000 (10 hours) | -| Path | `/` | -| HttpOnly | `true` | -| SameSite | `Strict` | -| Secure | `false` | - -### Token Validation - -Token validation includes caching — subsequent calls within the same request reuse the validated result: - -```go -// Cached in gin.Context: -// "validate-auth-token-object" → user data map -// "validate-auth-token-type" → token type string -// "validate-auth-token-err" → validation error -``` - ---- - -## Admin Tokens (Server-Level) - -Admin tokens provide token-based access for agents, CI/CD pipelines, and other automated systems without requiring the JWT login flow. - -### Configuration - -```yaml -api: - adminTokens: - - token: "agent-token-xyz" - label: "ci-deploy" # Optional: human-readable label for audit logs - permissions: - - project # Shorthand: all project CRUD operations - - project.reload # Reload project config - - user # Shorthand: all user CRUD operations - - server.shutdown - - namespace.myproject # All collections in myproject - - token: "readonly-token" - label: "monitoring" # Appears as tokenLabel in audit entries - permissions: - - project.read - - user.read - - namespace.*.*.read # Read-only across all namespaces and collections - - token: "importer-token" - label: "data-import" - permissions: - - namespace.myproject.articles.create # Only create in myproject/articles - - namespace.myproject.articles.update # Only update in myproject/articles -``` - -### Available Permissions - -| Permission | Description | -|-----------|-------------| -| `project` | Shorthand for all project operations (`project.*`) | -| `project.read` | GET projects | -| `project.create` | POST (create) projects | -| `project.update` | PUT (update) projects | -| `project.delete` | DELETE projects | -| `project.reload` | Reload project configuration | -| `user` | Shorthand for all user operations (`user.*`) | -| `user.read` | GET users | -| `user.create` | POST (create) users | -| `user.update` | PUT (update) users | -| `user.delete` | DELETE users | -| `server.shutdown` | Shutdown the server | -| `namespace` | Shorthand: all collections in all projects | -| `namespace.{project}` | Shorthand: all collections in a project | -| `namespace.{project}.{collection}` | Shorthand: all actions on a collection | -| `namespace.{project}.{collection}.read` | GET documents | -| `namespace.{project}.{collection}.create` | POST (create) documents | -| `namespace.{project}.{collection}.update` | PUT (update) documents | -| `namespace.{project}.{collection}.delete` | DELETE documents | - -### Wildcards - -Use `*` as a wildcard for any single segment in a permission string: - -| Permission | Description | -|-----------|-------------| -| `namespace.*` | All collections in any project | -| `namespace.*.articles.read` | Read articles across all projects | -| `namespace.myproject.*.read` | Read all collections in myproject | -| `namespace.*.*.read` | Read-only across everything | - -Shorthand works at any level — a permission with fewer segments implies all descendants: -- `namespace` = `namespace.*.*.*` (everything) -- `namespace.myproject` = `namespace.myproject.*.*` (all collections, all actions) -- `namespace.myproject.articles` = `namespace.myproject.articles.*` (all actions) - -### Usage - -Send the token via `X-Admin-Token` header or `Authorization: Bearer` header: - -``` -# Via X-Admin-Token header — project management -POST /api/v1/project -X-Admin-Token: agent-token-xyz -Content-Type: application/json - -{"name": "my-project", "description": "A new project", "configFile": "/path/to/config.yml"} -``` - -``` -# Via Authorization: Bearer header — collection access -GET /api/v1/_/myproject/articles -Authorization: Bearer readonly-token -``` - -``` -# Create document in a collection -POST /api/v1/_/myproject/articles -X-Admin-Token: importer-token -Content-Type: application/json - -{"title": "New Article", "body": "Content..."} -``` - -### Token Priority - -For project/user endpoints: -1. **Admin Token** (`X-Admin-Token` / `Authorization: Bearer`): Checked first. If present and valid, access is granted immediately. If present but invalid, access is denied (no fallthrough). -2. **JWT Auth** (`X-Auth-Token`): Checked as fallback if no admin token is provided. - -For collection endpoints (`/_/:project/:collection`): -1. **Admin Token**: If present, checked for `namespace.{project}.{collection}.{action}` permission. If present but lacking permission, access is denied (no fallthrough to JWT or public permissions). -2. **Collection Token Permissions**: Header/query-based tokens defined in collection config. -3. **Public / User / JWT**: Normal permission chain if no admin token is present. - -### Project Admin Endpoints - -For project admin endpoints (`reload`, `regenerate-search`, `backup`, `restore`), three auth methods are supported (checked in order): - -1. Server-level admin tokens with the corresponding permission (e.g. `project.reload`, `project.regenerate-search`, `project.backup`) -2. Project-level tokens configured in the project's `config.yml` (`admin.tokens[].allowReload`, `admin.tokens[].allowRegenerateSearch`, `admin.tokens[].allowBackup`) -3. JWT login with admin role (`X-Auth-Token` header) - -### Audit Trail - -All admin token actions are logged in the audit log with additional context: - -| Field | Description | Example | -|-------|-------------|---------| -| `authMethod` | Authentication method used | `admin-token`, `jwt` | -| `tokenLabel` | Label from token config (if set) | `ci-deploy` | -| `tokenPrefix` | First 8 characters of the token | `agent-to` | -| `ip` | Client IP address | `192.168.1.100` | - -The full token is **never** logged for security reasons. Use descriptive `label` values to identify tokens in audit logs. - ---- - -## Roles - -| Role | Value | Description | -|------|-------|-------------| -| Admin | `0` | Full access to all system and collection endpoints | -| User | `1` | Access limited to assigned projects and permitted methods | - -### Admin Role Behavior - -- Can access all projects regardless of assignment -- Can manage all users (create, update, delete) -- Can manage all projects -- Bypasses all collection permission checks -- No projection restrictions -- No pre-filter application -- Only admins can change `role`, `permissions`, and `llmBudgets` on users - -### User Role Behavior - -- Can only see projects belonging to their orgs (via `project.orgId` ∈ user's org IDs) -- Cannot create/update/delete projects (OrgAdmins can modify `teams[]`) -- Can only GET and PUT their own user profile (OrgAdmins have broader scope) -- Cannot change `username`, `role`, `permissions`, or `llmBudgets` -- Subject to collection permission checks (via team permissions) -- See [Orgs & Teams](18-orgs-teams.md) for the full role hierarchy - ---- - -## Permission System - -Each collection defines a `permissions` map with named permission sets: - -```yaml -permissions: - public: - methods: - get: true - filter: - published: true - user: - methods: - get: true - post: true - put: - allow: true - bulk: true # allow bulk PUT - delete: - allow: true - bulk: true # allow bulk DELETE - "token:apikey": - methods: - get: true - post: true - editor: - methods: - get: true - put: true # shorthand for {allow: true} -``` - -### PermissionSet Structure - -```go -type PermissionSet struct { - Methods map[string]MethodPermission // Allowed methods (get, post, put, delete) - Filter dbcrud.M // Auto-applied filter for queries - ValidProjections []string // Allowed projection names - CompiledFilterSubs []*CompiledSubst // Pre-compiled $subst expressions -} - -// MethodPermission represents a method permission. -// Can be unmarshalled from a boolean (true/false) or an object {allow: bool, bulk: bool}. -type MethodPermission struct { - Allow bool // Allow the method - Bulk bool // Allow bulk operations (POST array, PUT/DELETE without ID) -} -``` - -### Permission Check Priority - -When a request comes in, permissions are checked in this order: - -``` -1. Token-based permissions (header/query parameter match) - │ - ├── Token name found, value matches → Use token's permission set - ├── Token name found, value WRONG → ACCESS DENIED (no fallback!) - └── No token name found → Continue - │ -2. Public permissions - │ - ├── Method allowed → ACCESS GRANTED (no auth required) - └── Method not allowed → Continue - │ -3. Auth token validation (X-Auth-Token header) - │ - ├── No token / invalid → ACCESS DENIED - └── Valid token → Continue - │ -4. Admin role check (role = 0) - │ - ├── Is admin → ACCESS GRANTED - └── Not admin → Continue - │ -5. Team "user" permissions (role = 1, team-assigned to project) - │ - ├── User is member of a team assigned to this project - │ AND collection has "user" permission set → Use "user" set - └── Otherwise → Continue - │ -6. Custom user permissions - │ - ├── Auth permissions contain "namespace:permName" → Use that permission set - └── No match → Continue - │ -7. Custom team permissions - │ - ├── User's teams (assigned to project) have permissions matching a collection key - │ → Use that team permission set (union of all matching teams) - └── No match → ACCESS DENIED -``` - -See [Orgs & Teams — Permission System](18-orgs-teams.md#permission-system) for details on team-based permission resolution. - -### Token-Based Permissions - -Token permissions are matched against HTTP headers or query parameters: - -```yaml -permissions: - "token:secretkey123": # Matches ?token=secretkey123 OR header token: secretkey123 - methods: - get: true - "x-api-key:my-api-key": # Matches ?x-api-key=my-api-key OR header x-api-key: my-api-key - methods: - get: true - post: true -``` - -**Important:** The permission key format is `name:value`. The `name` part is checked against both query parameters and headers. If the `name` is found but the `value` doesn't match, access is **denied immediately** with no fallback to public/user permissions. - -### Custom Permissions - -Users can have custom permission strings in their `permissions` array: - -```json -{ - "username": "editor1", - "role": 1, - "permissions": ["myblog:editor", "myblog:moderator"] -} -``` - -These are matched against collection permission keys using the format `{namespace}:{permissionKey}`: - -```yaml -# In project config (namespace: myblog) -permissions: - editor: # Matched by "myblog:editor" - methods: - get: true - put: true - moderator: # Matched by "myblog:moderator" - methods: - get: true - delete: true -``` - ---- - -## Pre-Filters - -Permission sets can define filters that are automatically applied to all queries: - -```yaml -permissions: - public: - methods: - get: true - filter: - published: true - deletedAt: null - user: - methods: - get: true - post: true - put: true - filter: - author: - $subst: - expr: "auth.id" # Only see own documents -``` - -### How Pre-Filters Work - -1. The permission set's filter is deep-copied (to avoid mutation) -2. `$subst` expressions are resolved with the current request context -3. The filter is combined with any user-provided filter query via `$and` -4. Admin role (0) never has pre-filters applied - -### Filter Combination - -``` -Final filter = $and: [ - { _id: }, - { }, - { } -] -``` - ---- - -## Projection Restrictions - -Permission sets can restrict which projections are available: - -```yaml -permissions: - public: - methods: - get: true - validProjections: - - summary # Public can only use "summary" projection - user: - methods: - get: true - validProjections: - - summary - - detailed # Users can also use "detailed" -``` - -**Behavior:** - -- If `validProjections` is empty/not set → no restrictions (any projection or none) -- If `validProjections` is set and no `projection` parameter → first entry is used as default -- If `validProjections` is set and invalid projection requested → `400 Bad Request` -- Admin role (0) is never restricted - ---- - -## AuthData Model - -Extracted from the JWT token on each authenticated request: - -```go -type AuthData struct { - ID string // User ObjectID (hex) - Username string // Username - Role int // 0=Admin, 1=User - Permissions []string // Custom permission strings -} -``` - ---- - -## User Model - -```go -type User struct { - Model // ID, InsertTime, UpdateTime - Username *string // Required, alphanumeric, unique - Password *string // Required, min 3 chars (stored as MD5 hash) - Role *int // 0=Admin, 1=User (validated min=0, max=2) - Permissions []string // Custom permission strings - Meta map[string]interface{} // Arbitrary user metadata - LLMBudgets []LLMBudget // Per-provider LLM token budgets -} -``` - -### User Search - -The `_search` parameter on `/api/v1/user` searches across: -- `username` -- `meta.company` -- `meta.prename` -- `meta.surname` -- `meta.email` -- `meta.phone` - ---- - -## Server-Level Admin Authentication - -Server-level admin endpoints (e.g. shutdown) are protected by admin tokens with the appropriate permission: - -```bash -curl -X POST http://localhost:8080/api/v1/admin/shutdown \ - -H "X-Admin-Token: your-admin-token" -``` - -Admin tokens are configured via `api.adminTokens` in `config.yml`. See [Configuration](02-configuration.md) for details. - -## Personal Access Tokens (PAT) - -Users can create and manage their own API tokens. A PAT uses the same -`X-Auth-Token` header as the JWT and is always checked live against the owner's -current rights. Tokens are a core feature and always enabled. - -See [22-personal-access-tokens.md](22-personal-access-tokens.md) for format, -scope grammar, endpoints, examples and security properties. diff --git a/src/content/docs/server/06-hooks.md b/src/content/docs/server/06-hooks.md deleted file mode 100644 index 0bd04cd..0000000 --- a/src/content/docs/server/06-hooks.md +++ /dev/null @@ -1,1721 +0,0 @@ ---- -title: 'Hooks & JavaScript Runtime' ---- - -# Hooks & JavaScript Runtime - -## Overview - -Hooks are server-side JavaScript programs that execute at specific points in the collection CRUD lifecycle. They run in a **goja** runtime (ECMAScript 5.1+ with ES6 modules support) — **not** Node.js. There is no `require()`, no npm, no async/await. - -Each hook receives a `context` object containing the current request data and can modify it, abort the request, or perform side effects. - -## Configuration - -Hooks are defined in the project's `config.yml` under each collection: - -```yaml -collections: - - name: posts - hooks: - get: - read: - type: javascript - file: hooks/posts/get_read.js - timeout: 5000 # Hook timeout in milliseconds (optional) - ignoreRequestCancel: true # Continue even if the client disconnects (optional) - return: - type: javascript - file: hooks/posts/get_return.js - post: - bind: - type: javascript - file: hooks/posts/post_bind.js - validate: - type: javascript - file: hooks/posts/post_validate.js - create: - type: javascript - file: hooks/posts/post_create.js - return: - type: javascript - file: hooks/posts/post_return.js - put: - bind: - type: javascript - file: hooks/posts/put_bind.js - validate: - type: javascript - file: hooks/posts/put_validate.js - update: - type: javascript - file: hooks/posts/put_update.js - return: - type: javascript - file: hooks/posts/put_return.js - delete: - delete: - type: javascript - file: hooks/posts/delete.js - return: - type: javascript - file: hooks/posts/delete_return.js -``` - -### Hook Naming Convention - -Hooks are nested under `{method}` → `{step}`, each with a `HookConfig` (`type`, `file`, `timeout`, `ignoreRequestCancel`): - -| Method | Steps | -| -------- | ---------------------------------------------- | -| `get` | `read`, `return` | -| `post` | `bind`, `validate`, `create`, `return` | -| `post` | `bulkCreate`, `bulkReturn` (bulk mode) | -| `put` | `bind`, `validate`, `update`, `return` | -| `put` | `bulkUpdate`, `bulkReturn` (bulk mode) | -| `delete` | `delete`, `return` | -| `delete` | `bulkDelete`, `bulkReturn` (bulk mode) | -| `audit` | `return` | - -Hook files are resolved relative to the project's config directory. - -### Hook Execution Control - -Each hook can optionally control how long it may run and whether it should stop when the client disconnects. - -```yaml -hooks: - post: - create: - type: javascript - file: hooks/orders/post_create.js - timeout: 15000 - ignoreRequestCancel: true -``` - -- `timeout` is the maximum hook runtime in milliseconds. -- `ignoreRequestCancel: true` detaches the hook from the incoming HTTP request cancellation signal. Use this only for critical side effects that must finish even if the client disconnects. - -Important behavior: - -- `timeout` still applies even when `ignoreRequestCancel: true` is set. -- `ignoreRequestCancel` is intended for critical hooks such as order submission, payment capture, or writing to an outbox. -- For most business-critical flows, persisting work into a job or outbox is still the safer design than finishing everything synchronously inside the request hook. - -#### `DisableTimeout()` vs. `ignoreRequestCancel` - -These two controls solve different problems: - -- `ignoreRequestCancel` is a hook config option. It prevents the hook from being canceled when the HTTP client disconnects. -- `DisableTimeout()` is an internal runtime control used by built-in packages such as `channel.subscribe()` to remove the hook execution timeout for a specific long-running operation. - -`DisableTimeout()` is not a JavaScript API you call directly in a hook file. It only disables the execution timeout. It does **not** automatically ignore request cancellation unless the hook is also configured with `ignoreRequestCancel: true`. - -### Hook Lookup - -Hooks are looked up by key: `{collectionName}.{method}.{step}` - -Example: Collection `posts`, PUT request, update step → looks for key `posts.put.update` - ---- - -## Hook Context - -Each hook receives a `context` object with data specific to the method and step: - -### GET Hooks - -**`get.read`** - -```js -// context contains: -// context.filter — MongoDB filter (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**`get.return`** - -```js -// context contains: -// context.result — Array of documents (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -> **Count endpoint (`/_count`):** The `get.read` hook also runs for `GET /_count` requests, -> so filter modifications apply to the count query. The `get.return` hook does **not** run -> for `/_count` — there are no documents to transform. The hook receives `one: false`. - -### Audit Hooks - -**`audit.return`** - -Runs when audit log entries are served via `GET /api/v1/audit`. The hook is executed **per entry** for each audit log entry belonging to its collection. It runs for **all roles including admin** and affects entries retroactively (existing snapshots are filtered at read-time). - -Use this hook to strip sensitive fields (passwords, API keys, etc.) from audit snapshots that would otherwise be visible despite `get.return` hooks filtering them on normal collection reads. - -**Configuration:** - -```yaml -hooks: - audit: - return: - type: javascript - file: hooks/users/audit_return.js -``` - -**Hook context:** - -```js -// context contains: -// context.results — Array with a single audit log entry (mutable) -// context.one — Always true (entries are processed individually) -// context.auth() — Current authenticated user -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**Example — Remove passwords from audit snapshots:** - -```js -;(function () { - var results = context.results - if (results && results.length > 0) { - delete results[0].snapshot.password - delete results[0].snapshot.hashedPassword - } - return { results: results } -})() -``` - -**Behavior:** - -- Return `{ results: [modifiedEntry] }` to pass the modified entry through -- Return `{ results: [] }` to suppress the entry entirely (entry is removed from response) -- Throw an exception to abort the entire audit request -- If the hook does not set `results` in the return value, the original entry is passed through unchanged - -### POST Hooks - -**`post.bind`** - -Runs **once** at the beginning of a POST request, before body parsing and bulk detection. For bulk operations (array body), this hook is called exactly once, not per document. - -```js -// context contains: -// context.request().body() — Raw request body as string -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -// -// Return { data: [...] } to trigger bulk path with that array -// Return { data: {...} } to use that object as the document -// Return {} or nothing to let normal body parsing continue -``` - -**`post.validate`** - -```js -// context contains: -// context.data — Document to create, after bind (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**`post.create`** - -```js -// context contains: -// context.data — Document to create, after validation (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**`post.return`** - -```js -// context contains: -// context.data — Created document with ID (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -### PUT Hooks - -**`put.bind`** - -Runs **once** at the beginning of a PUT request, before body parsing. For bulk updates (PUT without ID), this hook is called exactly once, not per document. - -```js -// context contains: -// context.request().body() — Raw request body as string -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -// -// Return { data: {...} } to use that object as the update changes -// Return {} or nothing to let normal body parsing continue -``` - -**`put.validate`** - -```js -// context contains: -// context.data — Update data, after bind (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**`put.update`** - -```js -// context contains: -// context.data — Update data, after validation (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**`put.return`** - -```js -// context contains: -// context.data — Updated document (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -### DELETE Hooks - -**`delete.delete`** - -```js -// context contains: -// context.data — Document about to be deleted -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - -**`delete.return`** - -```js -// context contains: -// context.data — Deleted document -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -``` - ---- - -### Bulk Hooks (Optimised Path) - -When a `bulkUpdate` or `bulkDelete` hook step is defined — or `bulk.optimize` is enabled in the collection config — the server uses an optimised single-DB-call path instead of per-document processing. - -#### Bind Hook in Bulk Operations - -For both bulk and single-document operations, the `bind` hook runs **exactly once** at the beginning, **before** the request body is parsed. This applies to: - -- **POST with array body** (bulk create): `bind` runs once before bulk detection -- **PUT bulk** (without ID): `bind` runs once before body parsing - -The hook can return `data` as: -- **Array** → triggers bulk path with that array (POST only) -- **Object** → uses that object as the document/changes -- **Nothing** → normal body parsing continues - -This ensures consistent behavior and performance — the bind hook is never called N times for N documents. - -> **Important:** In the optimised path, the following per-document steps are **completely skipped:** -> -> | Skipped Step | Description | -> |---|---| -> | `validate` hook | Custom pre-validation logic | -> | Field validation | Type checking, `required`, `in`, `format`, `minLength`, `maxLength`, `min`, `max`, `minDate`, `maxDate`, `pattern`, `eval`, file validators | -> | `update` / `delete` hook | Per-document modification logic | -> | Per-document `return` hook | Individual result transformation | -> | Per-document audit | Individual audit entries per document | -> -> Instead, the server runs: **one** `bind` hook → **one** bulk hook (if defined) → **one** DB call → **one** audit entry → **one** `bulkReturn` hook (if defined). - -#### Activating the Optimised Path - -There are two ways to activate the optimised path: - -**1. Define a bulk hook step** (with custom logic): - -```yaml -hooks: - post: - bulkCreate: - type: javascript - file: hooks/products/bulk_create.js - bulkReturn: - type: javascript - file: hooks/products/bulk_create_return.js - put: - bulkUpdate: - type: javascript - file: hooks/products/bulk_update.js - bulkReturn: - type: javascript - file: hooks/products/bulk_update_return.js - delete: - bulkDelete: - type: javascript - file: hooks/products/bulk_delete.js - bulkReturn: - type: javascript - file: hooks/products/bulk_delete_return.js -``` - -**2. Use `bulk.optimize` config** (no hook file needed): - -```yaml -collections: - - name: products - bulk: - optimize: - create: true # POST with JSON array → optimised path - update: true # PUT without ID → optimised path - delete: true # DELETE without ID → optimised path -``` - -With `bulk.optimize`, the server goes straight to the DB operation without running any hook. This is the fastest option when no custom logic is needed. If both a bulk hook and `bulk.optimize` are set, the hook still runs. - -> **Warning — Dangers of the Optimised Path:** -> -> - **No field validation:** Required fields, type checks, `in`, `format`, `minLength`, `maxLength`, `min`, `max`, `minDate`, `maxDate`, `pattern`, `eval`, and file validators are not enforced. Invalid data can be written directly to the database. -> - **No per-document hooks:** If your `update` or `delete` hooks contain business logic (e.g. cascading deletes, notifications, cache invalidation), this logic will **not run**. -> - **No per-document audit trail:** Instead of N individual audit entries, only a single bulk entry with filter + count is logged. You lose the per-document snapshot of what changed. -> - **No file cleanup on bulk delete:** Upload directories for individual documents are still cleaned up, but only by collecting IDs beforehand — if this fails, orphaned files may remain. -> - **Operators bypass all safety:** `$inc`, `$mul`, `$unset` are sent directly to MongoDB without any application-level checks. - -**`post.bulkCreate`** - -```js -// context contains: -// context.data — Array of documents to create (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -// -// Return: { data: [...] } to modify the documents before validation and insert -``` - -**`put.bulkUpdate`** - -```js -// context contains: -// context.filter — MongoDB filter (mutable) -// context.data — Changes to apply (mutable, may contain operators like $inc, $mul) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -// -// Return: { filter: ..., data: ... } to modify filter/data before DB update -``` - -**`delete.bulkDelete`** - -```js -// context contains: -// context.filter — MongoDB filter (mutable) -// context.api — API config object -// context.project — Project config object -// context.collection — Collection config object -// -// Return: { filter: ... } to modify filter before DB delete -``` - -**`post.bulkReturn` / `put.bulkReturn` / `delete.bulkReturn`** - -```js -// context contains: -// context.filter — Filter that was used (bulkUpdate/bulkDelete only) -// context.data — Array of documents / changes (bulkCreate/bulkUpdate) -// context.created — Number of created documents (bulkCreate) -// context.ids — Array of created document IDs (bulkCreate) -// context.modified — Number of modified documents (bulkUpdate) -// context.deleted — Number of deleted documents (bulkDelete) -// -// Return: { results: ... } to override the response -``` - -> **Performance:** With `bulkUpdate`, the server runs the hook once, then executes a single `updateMany` call against the database. When field-level operators (e.g. `{"price": {"$mul": 1.1}}`) are used in the request body, they are converted to native MongoDB operators (`$inc`, `$mul`, `$unset`) and applied directly in the database — no documents are loaded into memory. This makes bulk updates with operators extremely fast, even for millions of documents. -> -> Without a bulk hook (per-document mode), operators are resolved in-memory per document before hooks/validation run, which is safer but slower. - -#### Empty Bulk Hook as Shortcut - -If you just want the optimised path without custom logic but prefer not to use `bulk.optimize`, you can define an empty hook file: - -```yaml -hooks: - put: - bulkUpdate: - type: javascript - file: hooks/empty.js # file contains no code -``` - -The mere existence of the `bulkUpdate` step triggers the optimised path. The `bulk.optimize` config is the cleaner alternative to this pattern. - ---- - -## HookException — Aborting with Custom Responses - -Throw an object to abort the request and send a custom response: - -```js -throw { - status: 400, // HTTP status code (default: 500) - msg: "Validation failed", // Error message - json: { error: "invalid_input" }, // JSON response body - html: "

Error

", // HTML response body - string: "plain text response", // Plain text response - bytes: someByteArray, // Raw bytes response - file: "/path/to/file.pdf" // File download response -}; -``` - -**Priority:** Only one response type is used. If multiple are set, priority is: `json` > `html` > `string` > `bytes` > `file`. - -**Important:** If `response.write()` has already been called (streaming started), throwing an exception will only abort the request — no additional response body can be sent. - -### Examples - -```js -// Return 403 with JSON -throw { status: 403, json: { error: "forbidden", reason: "Not your document" } }; - -// Return 404 with message -throw { status: 404, msg: "Document not found" }; - -// Return HTML page -throw { status: 200, html: "Custom page" }; - -// Return a file -throw { status: 200, file: "/data/exports/report.pdf" }; -``` - ---- - -## Timeout Configuration - -Timeout is configured per hook via the `timeout` field in the `HookConfig`: - -```yaml -hooks: - get: - read: - type: javascript - file: hooks/get_read.js - timeout: 5000 # milliseconds -``` - -- When timeout expires, the goja runtime is interrupted -- `DisableTimeout()` can be called programmatically (e.g., by `channel.subscribe`) -- Jobs and validators also use this timeout mechanism - ---- - -## Context Packages Reference - -All packages are available as `{package}.{method}()` in hook JavaScript code. - -### `db` — Database Operations - -```js -// Find documents -var results = context.db.find("collectionName", { - filter: { status: "active" }, - selector: { title: 1, body: 1 }, // Field selection - projection: "summary", // OR named projection - offset: 0, - limit: 10, - sort: ["-createdAt", "title"] // Prefix "-" for descending -}); - -// Count documents -var count = context.db.count("collectionName", { - filter: { status: "active" } -}); - -// Create document -var created = context.db.create("collectionName", { - title: "New Post", - body: "Content here" -}); -// Returns: { id: "...", title: "New Post", ... } - -// Update document -var updated = context.db.update("collectionName", "60a1b2c3d4e5f6g7h8i9j0k1", { - title: "Updated Title" -}); - -// Update document with field-level operators (atomic — single DB call) -var updated = context.db.update("collectionName", "60a1b2c3d4e5f6g7h8i9j0k1", { - price: { "$mul": 1.1 }, // multiply current price by 1.1 - stock: { "$inc": -1 }, // decrement stock by 1 - title: "Also update title" // plain value, sent via $set -}); -// Operators are converted to native MongoDB operators and executed atomically. -// No pre-read needed — the database handles the computation. - -// Update document with raw MongoDB update (advanced) -var updated = context.db.update("collectionName", "60a1b2c3d4e5f6g7h8i9j0k1", { - "$set": { title: "New" }, - "$inc": { counter: 1 } -}, { raw: true }); -// With { raw: true }, the data is passed directly to MongoDB as an update document. -// Use this when you need the full MongoDB update syntax (e.g. $rename, $push). - -// Delete document -context.db.delete("collectionName", "60a1b2c3d4e5f6g7h8i9j0k1"); -// Returns: { message: "ok" } - -// Delete many documents -var result = context.db.deleteMany("collectionName", { - filter: { status: "archived" } -}); -// Returns: { message: "ok", removed: 5 } - -// Update many documents (simple $set) -var result = context.db.updateMany("collectionName", { - filter: { category: "sale" }, - changes: { discount: 10 } -}); -// Returns: { message: "ok", modified: 12 } - -// Update many documents — changes with field-level operators -var result = context.db.updateMany("collectionName", { - filter: { category: "sale" }, - changes: { - price: { "$mul": 0.9 }, // 10% discount - stock: { "$inc": -1 }, // decrement stock - onSale: true // plain value — mixed with operators - } -}); -// Operators in changes are auto-detected and converted to native MongoDB operators. -// Returns: { message: "ok", modified: 12 } - -// Update many documents (raw MongoDB operators for optimised DB operations) -var result = context.db.updateMany("collectionName", { - filter: { category: "electronics" }, - update: { - "$inc": { stock: -1 }, - "$mul": { price: 1.1 }, - "$set": { lastUpdated: "2024-01-01" } - } -}); -// Returns: { message: "ok", modified: 8 } -``` - -**Special filter values:** - -- `{ "$date": "2024-01-15T00:00:00Z" }` — Converted to Go `time.Time` -- Supports all standard MongoDB operators except `$where`, `$accumulator`, `$function`, `$let`, `$setField` - -**Audit:** All write operations (`create`, `update`, `delete`, `deleteMany`, `updateMany`) automatically write audit log entries. `deleteMany` and `updateMany` are audited as `delete` and `update` respectively. - ---- - -### `http` — HTTP Requests - -```js -// Simple GET -var resp = context.http.fetch("https://api.example.com/data"); -var data = resp.body.json(); -var text = resp.body.text(); - -// POST with headers -var resp = context.http.fetch("https://api.example.com/webhook", { - method: "POST", - headers: { - "Content-Type": "application/json", - "Authorization": "Bearer token123" - }, - body: context.json.stringify({ event: "created" }), - timeout: 30000 // milliseconds (default: 10000) -}); - -// Check response -if (resp.status !== 200) { - throw { status: 502, json: { error: "upstream_error", status: resp.status } }; -} - -// Streaming response (line-by-line) -var stream = http.fetchStream("https://api.example.com/stream", { - method: "GET", - timeout: 0 // no timeout (default for streams) -}); - -var line; -while ((line = stream.body.read()) !== null) { - // Process each line - context.response.writeSSE("data", line); -} -stream.body.close(); -``` - -**Response object:** - -```js -{ - headers: {}, // Original + lowercased keys - status: 200, - statusText: "OK", - trailer: {}, - url: "https://...", - body: { - text(): string, // Full body text - json(): object // Parsed JSON (throws on error) - } -} -``` - -**Stream response:** - -```js -{ - headers: {}, - status: 200, - statusText: "OK", - body: { - read(): string|null, // Next line, null at EOF - close(): void // Close response body - } -} -``` - ---- - -### `fs` — Filesystem - -```js -// Get config directory path -var dir = context.fs.configDir(); - -// Read file as string -var content = context.fs.readFile("/path/to/file.txt"); - -// Read file as bytes -var bytes = context.fs.readFile("/path/to/file.bin", { bytes: true }); - -// Write file (string or bytes) -context.fs.writeFile("/path/to/output.txt", "Hello World"); -context.fs.writeFile("/path/to/output.bin", bytes); - -// File info -var info = fs.stat("/path/to/file.txt"); -// { name: "file.txt", isDir: false, size: 1234, modTime: "..." } - -// List directory -var entries = fs.readDir("/path/to/dir"); -// [{ name: "file.txt", isDir: false, size: 1234, modTime: "..." }, ...] - -// Create directory (recursive) -context.fs.mkDir("/path/to/new/dir"); - -// Remove file or empty directory -context.fs.remove("/path/to/file.txt"); - -// Copy file or directory -context.fs.copy("/path/to/source", "/path/to/destination"); -``` - -**Notes:** - -- Paths are sanitized via `CleanFilePath` -- File operations execute under the project's configured UID (suexec) -- Permissions: `0755` for new files and directories - ---- - -### `smtp` — Email - -```js -context.smtp.sendMail({ - to: "user@example.com", // string or string[] - cc: ["cc1@example.com"], // optional - bcc: "bcc@example.com", // optional - from: "noreply@example.com", // required - fromName: "My App", // optional - replyTo: "support@example.com", // optional - subject: "Welcome!", - plain: "Welcome to our app.", // plain text body - html: "

Welcome!

", // HTML body - attach: ["/path/to/file.pdf"], // file paths for attachments - - // Optional: Override server SMTP config - host: "smtp.custom.com:587", - username: "user", - password: "pass" -}); -``` - ---- - -### `jwt` — JSON Web Tokens - -```js -// Create a signed JWT -var token = context.jwt.create( - { userId: "123", role: "admin" }, // claims - { - secret: "my-secret-key", // optional (random temp secret if omitted) - validityDuration: 3600 // seconds → sets "exp" claim - } -); - -// Parse and verify a JWT -var result = context.jwt.parse(token, { secret: "my-secret-key" }); -// { -// valid: true, -// method: { ... }, -// header: { alg: "HS256", typ: "JWT" }, -// claims: { userId: "123", role: "admin", exp: 1234567890 } -// } - -// Invalid token -var result = context.jwt.parse("bad-token", { secret: "wrong" }); -// { valid: false, error: "signature is invalid" } -``` - -**Notes:** - -- Only HMAC signing methods are accepted -- If no `secret` is provided, a random 16-byte temporary secret (generated at server start) is used - ---- - -### `bcrypt` — Password Hashing - -```js -// Hash a password -var hash = context.bcrypt.hash("mypassword"); -// With custom cost -var hash = context.bcrypt.hash("mypassword", { cost: 12 }); // default: 14 - -// Verify a password -var valid = context.bcrypt.check("mypassword", hash); // true or false -``` - ---- - -### `crypto` — Cryptographic Hashes - -```js -var sha = context.crypto.sha256("hello world"); -// "b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9" - -var md5 = context.crypto.md5("hello world"); -// "5eb63bbbe01eeed093cb22bb8f5acdc3" -``` - ---- - -### `base64` — Encoding/Decoding - -```js -// Encode string -var encoded = context.base64.encode("Hello World"); -// "SGVsbG8gV29ybGQ=" - -// Encode bytes (from fs.readFile with bytes: true) -var encoded = context.base64.encode(context.fs.readFile("/path/to/file", { bytes: true })); - -// Decode to string -var decoded = context.base64.decode("SGVsbG8gV29ybGQ="); -// "Hello World" - -// Decode to bytes -var bytes = context.base64.decode("SGVsbG8gV29ybGQ=", { returnBytes: true }); -``` - ---- - -### `json` — JSON Parsing/Serialization - -```js -var obj = json.parse('{"key": "value"}'); -var str = context.json.stringify({ key: "value" }); -``` - ---- - -### `xml` — XML Creation/Parsing - -```js -// Create XML -var xmlStr = xml.create({ - person: { - name: "John", - age: 30 - } -}); -// -// John -// 30 -// - -// With options -var xmlStr = xml.create( - { name: "John", age: 30 }, - { - rootElement: "person", - includeHeader: true - } -); -// -// -// John -// 30 -// - -// Parse XML -var obj = context.xml.parse('John30'); -// { person: { name: "John", age: "30" } } - -// Parse from bytes -var obj = context.xml.parse(context.fs.readFile("/path/to/file.xml", { bytes: true })); -``` - -**Notes:** XML parsing supports automatic charset detection. - ---- - -### `tpl` — Template Engine (Pongo2/Django-Style) - -```js -var html = context.tpl.execute( - "Hello {{ name }}! You have {{ count }} messages.", - { name: "John", count: 5 } -); -// "Hello John! You have 5 messages." - -// With logic -var result = context.tpl.execute( - "{% for item in items %}- {{ item.title }}\n{% endfor %}", - { items: [{ title: "A" }, { title: "B" }] } -); - -// Template functions are supported -var result = context.tpl.execute( - "{{ greeting() }}", - { greeting: function() { return "Hello!"; } } -); -``` - -Uses the [pongo2](https://github.com/flosch/pongo2) template engine (Django/Jinja2-compatible syntax). - ---- - -### `image` — Image Processing - -```js -context.image.filter("/path/to/source.jpg", "/path/to/output.webp", [ - { - width: 800, - height: 600, - fit: true, // Fit within dimensions (maintains aspect ratio) - // fill: true, // Fill dimensions (crop to fit) - resampling: "lanczos", // Resampling algorithm - anchor: "center", // Crop anchor for fill mode - brightness: 10, // -100 to 100 - saturation: 0, // -100 to 100 - contrast: 5, // -100 to 100 - gamma: 1.0, // > 0 - blur: 0, // Gaussian blur sigma - sharpen: 0, // Unsharp mask sigma - invert: false, - grayscale: false, - quality: 80, // JPEG/WebP quality (1-100) - lossless: false, // WebP lossless mode - outputType: "webp", // "jpeg", "png", "webp", "gif", "bmp", "tiff" - skipLargerDimension: true, // Skip if result would be larger - skipLargerFilesize: true // Skip if file size would increase - } -]); -``` - -**Resampling algorithms:** `nearestNeighbor`, `box`, `linear`, `hermite`, `mitchellNetravali`, `catmullRom`, `bSpline`, `gaussian`, `bartlett`, `lanczos`, `hann`, `hamming`, `blackman`, `welch`, `cosine` - -**Anchor positions:** `center`, `topLeft`, `top`, `topRight`, `left`, `right`, `bottomLeft`, `bottom`, `bottomRight` - ---- - -### `pdf` — PDF Generation & Manipulation - -```js -// Generate PDF from HTML -var pdfBytes = context.pdf.fromHTML("

Report

", { - orientation: "Portrait", // "Portrait" or "Landscape" - marginTop: 10, // millimeters - marginBottom: 10, - marginLeft: 15, - marginRight: 15, - dpi: 300, - title: "My Report", - printMediaType: true, - headerHTML: "
Header
", - footerHTML: "
Page [page] of [topage]
", - footerFontSize: 8 -}); - -// Write PDF to file -context.fs.writeFile("/path/to/report.pdf", pdfBytes); - -// Or return directly -throw { status: 200, bytes: pdfBytes }; -``` - -**PDF manipulation with pdfcpu:** - -```js -// Merge multiple PDFs -var merged = context.pdf.cpu("merge", [pdf1Bytes, pdf2Bytes]); - -// Rotate pages -var rotated = context.pdf.cpu("rotate", pdfBytes, { rotation: 90 }); - -// Add text watermark -var watermarked = context.pdf.cpu("watermark", pdfBytes, { - mode: "text", - text: "DRAFT", - pages: ["1-3"] -}); - -// Add image watermark -var stamped = context.pdf.cpu("stamp", pdfBytes, { - mode: "image", - bytes: imageBytes -}); - -// Add PDF overlay -var overlaid = context.pdf.cpu("stamp", pdfBytes, { - mode: "pdf", - file: "/path/to/overlay.pdf" -}); - -// Create new PDF with metadata -var newPdf = context.pdf.cpu("create", null, { - description: { title: "Test", author: "System" } -}); -``` - -**Commands:** `create`, `rotate`, `merge`, `stamp`, `watermark` - ---- - -### `cookie` — Cookie Management - -> **Only available in hooks** (not in jobs). - -```js -// Get cookie value -var sessionId = cookie.get("session_id"); - -// Set cookie -context.cookie.set("session_id", "abc123", { - maxAge: 86400, // seconds - path: "/", - domain: "example.com", - httpOnly: true, - secure: true, - sameSite: "strict" // "strict", "lax", "none" -}); -``` - ---- - -### `user` — Authentication Info - -> **Only available in hooks** (not in jobs). - -```js -var authUser = user.auth(); -if (authUser) { - // { - // id: "60a1b2c3d4...", - // username: "admin", - // role: "0", - // permissions: ["blog:editor"] - // } -} else { - // Not authenticated -} -``` - ---- - -### `config` — Configuration Access - -```js -var ns = context.config.namespace(); -// "myblog" - -var projectConfig = context.config.project(); -// Full project configuration as object - -var serverConfig = context.config.server(); -// Full server configuration as object -``` - ---- - -### `debug` — Debugging Tools - -```js -// Print to server stdout (uses spew.Dump for deep inspection) -context.debug.dump(someObject, "label", 42); - -// Get Sentry trace ID (only in hooks) -var traceId = debug.sentryTraceId(); - -// Sleep (blocking) -context.debug.sleep(1000); // milliseconds -``` - ---- - -### `exec` — Shell Command Execution - -```js -// Simple command -var output = context.exec.command("echo", { args: ["hello"] }); -// "hello\n" - -// With options -var result = context.exec.command("python3", { - args: ["script.py"], - stdin: "input data", - dir: "/working/directory", - env: ["KEY=value"], - timeout: 30000, // milliseconds (default: 10000) - returnObject: true, - combineOutput: false -}); -// { stdout: "output", stderr: "", exitCode: 0 } - -// Combined output -var result = context.exec.command("make", { - args: ["build"], - returnObject: true, - combineOutput: true -}); -// { output: "combined stdout+stderr", exitCode: 0 } -``` - -**Notes:** - -- Default working directory is the project's config directory -- Executes under project's configured UID (suexec) -- Exit code != 0 throws error (unless `returnObject: true`) - ---- - -### `response` — HTTP Response Control - -> **Only available in hooks** (not in jobs). - -```js -// Set response header -context.response.header("X-Custom-Header", "value"); -context.response.header("Content-Type", "text/html"); - -// Set status code (must be before write) -context.response.writeHeader(200); - -// Write to response body -context.response.write("Hello"); - -// Flush (for streaming) -context.response.flush(); - -// Server-Sent Events helper -context.response.writeSSE("event-name", context.json.stringify({ data: "value" })); -// Sends: "event: event-name\ndata: {...}\n\n" and flushes - -// SSE without event name -context.response.writeSSE(context.json.stringify({ data: "value" })); -// Sends: "data: {...}\n\n" and flushes -``` - -**Streaming pattern:** - -```js -context.response.header("Content-Type", "text/event-stream"); -context.response.header("Cache-Control", "no-cache"); -context.response.header("Connection", "keep-alive"); -context.response.writeHeader(200); - -for (var i = 0; i < 10; i++) { - context.response.writeSSE("tick", context.json.stringify({ count: i })); - context.debug.sleep(1000); -} - -throw { status: 200 }; // Signal end (status is ignored since headers already sent) -``` - ---- - -### `cache` — Project-Scoped Memory Cache - -A key-value in-memory cache scoped to the current project. Data persists across requests but is cleared on project reload. Supports TTL (time-to-live) and LRU eviction when the max entry limit is reached. - -```js -// Store a value (TTL in milliseconds, optional — 0 uses server default) -context.cache.set("user:123", { name: "Alice", role: "admin" }, 300000); // 5 min TTL -context.cache.set("config", loadExpensiveConfig()); // no TTL (or server default) - -// Retrieve a value (returns null if missing or expired) -var user = context.cache.get("user:123"); -if (user !== null) { - // cache hit -} - -// get() hands out a COPY — mutating it does not update the cached entry. -// Write it back explicitly with set() if you want the change to persist. -var cfg = context.cache.get("config") || { retries: 0 }; -cfg.retries += 1; -context.cache.set("config", cfg); - -// Check existence -if (cache.has("user:123")) { /* ... */ } - -// Delete a single key (returns true if it existed) -context.cache.delete("user:123"); - -// Clear all entries for this project -context.cache.clear(); - -// Get the number of live (non-expired) entries -var n = cache.count(); -``` - -**Configuration** (server `config.yml`): - -```yaml -hook: - cache: - maxEntries: 10000 # Max entries before LRU eviction (default: 10000) - defaultTTL: "5m" # Default TTL when cache.set() is called with ttl=0 (default: no expiry) -``` - -Per-project override in project `config.yml`: - -```yaml -hook: - cache: - maxEntries: 5000 # Override server default for this project - defaultTTL: "10m" # Override default TTL for this project -``` - -**Notes:** - -- The cache is **project-scoped** — each project has its own isolated cache instance -- Data is stored in-memory only (not persisted to disk or database) -- On project reload, the cache is cleared and re-created with current config -- Values can be any JSON-serializable type (strings, numbers, objects, arrays) -- A background goroutine periodically removes expired entries -- `get()` returns a **deep copy**, and `set()` stores a deep copy of what you pass in. A cached value is never shared between concurrent requests, so mutating a `get()` result stays local to that request — call `set()` to persist it. Without this isolation, two requests mutating the same cached object while one of them uses it in a query crashed the server with `fatal error: concurrent map iteration and map write`. - ---- - -### `ratelimit` — Rate Limiting Helpers - -The `ratelimit` package provides two complementary strategies to protect your application from abuse. Both are **project-scoped** (each project has its own isolated state) and persist across hook invocations in memory. - -| Strategy | Purpose | How it works | Typical use case | -| --- | --- | --- | --- | -| **Exponential Backoff** | Punish repeated failures | Each failure doubles the wait time before the next attempt is allowed (1 s → 2 s → 4 s → … up to a cap). After a quiet period the counter resets. | Login brute-force protection, OTP verification | -| **Sliding Window** | Cap throughput | Counts how many hits a key has produced within a rolling time window. Once the limit is reached, further requests are rejected until older hits fall out of the window. | API call quotas, upload limits | - -> **When to use which?** -> -> - Use **backoff** when every failed attempt should make the _next_ attempt harder (progressive punishment). -> - Use **window** when you simply want to allow a fixed number of actions per time period (flat quota). -> - You can combine both on different keys in the same hook. - ---- - -#### Exponential Backoff - -After the first failure, the key is blocked for an _initial delay_ (default 1 s). Each subsequent failure doubles the delay (2 s → 4 s → 8 s → …) until a _maximum delay_ cap is reached (default 60 s). If no further failure is recorded for the _reset period_ (default 15 min), the backoff counter returns to zero. - -```text -Failure #1 → blocked 1 s -Failure #2 → blocked 2 s -Failure #3 → blocked 4 s - ... ... -Failure #N → blocked min(2^(N-1) s, maxDelay) - (no failure for resetAfter) → counter resets to 0 -``` - -```js -var ip = request().ip; -var key = "login:" + ip; - -// Check if the key is currently blocked -var result = ratelimit.backoffCheck(key); -// result = { blocked: false, retryAfterMs: 0 } -// result = { blocked: true, retryAfterMs: 3500 } - -if (result.blocked) { - throw { - status: 429, - json: { - error: "too_many_attempts", - retryAfterMs: result.retryAfterMs - } - }; -} - -// Record a failure (increases backoff delay) -context.ratelimit.backoffRecord(key); - -// Reset after successful action (e.g. successful login) -context.ratelimit.backoffReset(key); -``` - -| Function | Description | -| --- | --- | -| `backoffCheck(key)` | Returns `{ blocked: bool, retryAfterMs: int }` — check _before_ processing | -| `backoffRecord(key)` | Record a failure — increases the delay for future checks | -| `backoffReset(key)` | Clear all backoff state for the key (e.g. after a successful login) | - ---- - -#### Sliding Window - -A sliding window tracks how many times a key was used within a rolling time period (default 60 s). As long as the count stays below the configured maximum (default 60), the request is allowed. Once the limit is hit, the caller must wait until older entries slide out of the window. - -```text -Window = 60 s, maxHits = 5 - -Time 0 s 10 s 20 s 30 s 40 s 50 s 60 s 70 s -Hits #1 #2 #3 #4 #5 ✗(!) #6 #7 - │ │ - limit reached #1 slid out → space freed -``` - -```js -var userId = user.auth().id; -var key = "api:" + userId; - -// Record a hit and check if within limit -var result = ratelimit.windowCheck(key); -// result = { allowed: true, retryAfterMs: 0 } -// result = { allowed: false, retryAfterMs: 12000 } - -if (!result.allowed) { - throw { - status: 429, - json: { - error: "rate_limit_exceeded", - retryAfterMs: result.retryAfterMs - } - }; -} - -// Get current hit count in the window -var count = ratelimit.windowCount(key); - -// Reset the window for a key -context.ratelimit.windowReset(key); -``` - -| Function | Description | -| --- | --- | -| `windowCheck(key)` | Record a hit and return `{ allowed: bool, retryAfterMs: int }` | -| `windowCount(key)` | Return the current number of hits in the window (read-only) | -| `windowReset(key)` | Clear all hits for the key | - ---- - -#### Configuration - -Server `config.yml` (defaults for all projects): - -```yaml -hook: - ratelimit: - backoffInitialDelay: "1s" # Delay after first failure (default: 1s) - backoffMaxDelay: "60s" # Maximum backoff cap (default: 60s) - backoffResetAfter: "15m" # Reset counter after this idle period (default: 15m) - windowSize: "60s" # Sliding window duration (default: 60s) - windowMaxHits: 60 # Max hits per window (default: 60) -``` - -Project-level override (project `config.yml`) — only the fields you specify are overridden, the rest fall back to server defaults: - -```yaml -hook: - ratelimit: - backoffInitialDelay: "2s" # Stricter backoff for this project - windowMaxHits: 30 # Lower quota for this project - windowSize: "30s" -``` - -#### Notes - -- Both strategies are **project-scoped** — each project has its own limiter instance -- State is in-memory and cleared on project reload -- Keys are arbitrary strings — use meaningful prefixes (e.g. `"login:"`, `"api:"`, `"upload:"`) -- A background goroutine periodically cleans up stale entries -- All five rate-limit parameters can be overridden per project via `hook.ratelimit` in the project config - ---- - -### `channel` — Realtime Pub/Sub - -> **Only available in hooks** (not in jobs). - -```js -// Subscribe to a channel (blocking) -var sub = context.channel.subscribe("chat-room", { - bufferSize: 100, - onFull: "drop-newest", - messageTTL: 60000, // ms — message expiry - lastN: 10, // Receive last N messages on connect - maxAge: 30000 // ms — only replay messages newer than this -}); - -// Receive loop -context.response.header("Content-Type", "text/event-stream"); -context.response.writeHeader(200); - -var msg; -while ((msg = sub.receive()) !== null) { - context.response.writeSSE("message", context.json.stringify(msg)); -} -sub.close(); - -// Send to a channel (from any hook) -context.channel.send("chat-room", { - user: "john", - text: "Hello everyone!" -}); -``` - -**Notes:** - -- `subscribe()` automatically disables the hook timeout -- `receive()` blocks until a message arrives or the client disconnects (returns `null`) -- Cleanup is automatically registered (subscription is closed when hook ends) -- The channel uses a ring buffer — see [07-realtime.md](07-realtime.md) for architecture details - ---- - -### `charset` — Character Encoding - -```js -// Decode bytes as UTF-8 -var utf8String = context.charset.decode(bytes, "utf-8"); - -// Decode bytes as ISO-8859-1 -var latin1String = context.charset.decode(bytes, "iso-8859-1"); - -// Convert ISO-8859-1 to UTF-8 -var utf8String = context.charset.iso8859ToUtf8(isoBytes); - -// Also accepts string or byte arrays -var utf8String = context.charset.iso8859ToUtf8("iso string"); -``` - -Supported encodings for `context.charset.decode(...)`: - -- `utf-8` / `utf8` -- `iso-8859-1` / `iso8859-1` / `latin1` -- `windows-1252` -- `iso-8859-15` -- `utf-16le` / `utf-16be` -- `shift-jis` / `shift_jis` - -Additionally, many common IANA/HTML encoding labels supported by `golang.org/x/text/encoding/htmlindex` can be used. - -Behavior: - -- accepts strings, `Int8Array`, and numeric byte arrays -- removes a leading UTF-8 BOM when decoding UTF-8 -- returns an error for invalid UTF-8 byte sequences - ---- - -### `request` — HTTP Request Data - -> **Only available in hooks** (not in jobs). Registered as a top-level function, not a package. - -```js -var req = request(); - -req.method; // "get", "post", etc. (lowercase) -req.remoteAddr; // "192.168.1.1:54321" -req.clientIp; // "192.168.1.1" -req.host; // "example.com" -req.url; // "/api/v1/ns/posts?limit=10" -req.path; // "/api/v1/ns/posts" - -// URL route parameters -var id = req.param("id"); - -// Query parameters -var limit = req.query("limit"); - -// Request headers -var auth = req.header("Authorization"); - -// Request body -var bodyText = req.body(); -var bodyBytes = req.bodyBytes(); -``` - ---- - -## Legacy Compatibility - -All original function names are still available as top-level functions. New code should use the package-based API. - -| Legacy Name | New API | Description | -| -------------------------- | ---------------------------- | ------------------- | -| `readCollection(...)` | `db.find(...)` | Find documents | -| `readCollectionCount(...)` | `db.count(...)` | Count documents | -| `createDocument(...)` | `db.create(...)` | Create document | -| `updateDocument(...)` | `db.update(...)` | Update document | -| `deleteDocument(...)` | `db.delete(...)` | Delete document | -| `deleteDocuments(...)` | `db.deleteMany(...)` | Delete many | -| — | `db.updateMany(...)` | Update many (new) | -| `mail(...)` | `smtp.sendMail(...)` | Send email | -| `template(...)` | `context.tpl.execute(...)` | Render template | -| `file(...)` | `fs.readFile(...)` | Read file | -| `fetch(...)` | `http.fetch(...)` | HTTP request | -| `dump(...)` | `context.debug.dump(...)` | Debug output | -| `header(...)` | `response.header(...)` | Set response header | -| `auth()` | `user.auth()` | Get auth data | -| `sentryTraceId()` | `debug.sentryTraceId()` | Sentry trace ID | -| `bcryptHash(...)` | `context.bcrypt.hash(...)` | Hash password | -| `bcryptCheck(...)` | `context.bcrypt.check(...)` | Check password | -| `jwtCreate(...)` | `context.jwt.create(...)` | Create JWT | -| `jwtParse(...)` | `context.jwt.parse(...)` | Parse JWT | -| `iso8859ToUtf8(...)` | `context.charset.iso8859ToUtf8(...)` | Charset conversion | -| `fsStat(...)` | `fs.stat(...)` | File info | -| `fsReadDir(...)` | `fs.readDir(...)` | List directory | -| `fsMkDir(...)` | `fs.mkDir(...)` | Create directory | -| `imageFilter(...)` | `image.filter(...)` | Process image | -| `xmlCreate(...)` | `xml.create(...)` | Create XML | -| `xmlParse(...)` | `context.xml.parse(...)` | Parse XML | -| `cookieGet(...)` | `cookie.get(...)` | Get cookie | -| `cookieSet(...)` | `cookie.set(...)` | Set cookie | - ---- - -## TypeScript Definitions - -The `tibi-types` package provides TypeScript type definitions for all hook context packages. While hooks run in a non-TypeScript runtime, these types enable IDE autocompletion and type checking during development. - -See [tibi-types/](../tibi-types/) for available type definitions. - ---- - -## Practical Examples - -### Validate and Transform Before Create - -```js -// hooks/posts/before_create.js -var auth = user.auth(); -if (!auth) { - throw { status: 401, json: { error: "unauthorized" } }; -} - -// Set author from authenticated user -context.data.author = auth.id; -context.data.authorName = auth.username; -context.data.status = "draft"; -context.data.slug = context.data.title - .toLowerCase() - .replace(/[^a-z0-9]+/g, "-") - .replace(/^-|-$/g, ""); - -// Validate uniqueness -var existing = context.db.find("posts", { - filter: { slug: context.data.slug }, - limit: 1 -}); -if (existing.length > 0) { - throw { status: 409, json: { error: "slug_exists", slug: context.data.slug } }; -} -``` - -### Send Notification After Create - -```js -// hooks/posts/after_create.js -var auth = user.auth(); - -// Notify admin -context.smtp.sendMail({ - to: "admin@example.com", - from: "noreply@example.com", - subject: "New post: " + context.data.title, - html: context.tpl.execute( - fs.readFile(context.fs.configDir() + "/templates/new_post_email.html"), - { post: context.data, author: auth } - ) -}); - -// Notify realtime subscribers -context.channel.send("posts-feed", { - type: "new_post", - id: context.data.id, - title: context.data.title -}); -``` - -### SSE Streaming Endpoint - -```js -// hooks/events/get.before -context.response.header("Content-Type", "text/event-stream"); -context.response.header("Cache-Control", "no-cache"); -context.response.header("Connection", "keep-alive"); -context.response.writeHeader(200); - -var sub = context.channel.subscribe("events", { - bufferSize: 50, - lastN: 5 -}); - -var msg; -while ((msg = sub.receive()) !== null) { - context.response.writeSSE("event", context.json.stringify(msg)); -} - -sub.close(); -throw { status: 200 }; -``` - -### Proxy External API with Caching - -```js -// hooks/weather/get.before -var req = request(); -var city = req.query("city"); -if (!city) { - throw { status: 400, json: { error: "city parameter required" } }; -} - -// Check cache -var cached = context.db.find("_weather_cache", { - filter: { - city: city, - fetchedAt: { "$gte": { "$date": new Date(Date.now() - 3600000).toISOString() } } - }, - limit: 1 -}); - -var data; -if (cached.length > 0) { - data = cached[0].data; -} else { - var resp = context.http.fetch("https://api.weather.example.com/v1/current?city=" + city, { - headers: { "Authorization": "Bearer " + context.config.project().weatherApiKey }, - timeout: 5000 // milliseconds (default: 10000) - }); - data = resp.body.json(); - - context.db.create("_weather_cache", { - city: city, - data: data, - fetchedAt: new Date().toISOString() - }); -} - -throw { status: 200, json: data }; -``` - -### PDF Report Generation - -```js -// hooks/reports/get.before -var req = request(); -var reportId = req.param("id"); - -var report = context.db.find("reports", { - filter: { id: reportId }, - limit: 1 -}); - -if (report.length === 0) { - throw { status: 404, json: { error: "report_not_found" } }; -} - -var html = context.tpl.execute( - fs.readFile(context.fs.configDir() + "/templates/report.html"), - { report: report[0] } -); - -var pdfBytes = context.pdf.fromHTML(html, { - orientation: "Portrait", - marginTop: 15, - marginBottom: 15, - marginLeft: 20, - marginRight: 20, - footerHTML: "
Page [page]/[topage]
", - footerSpacing: 5 -}); - -context.response.header("Content-Type", "application/pdf"); -context.response.header("Content-Disposition", 'attachment; filename="report-' + reportId + '.pdf"'); -context.response.writeHeader(200); -context.response.write(pdfBytes); - -throw { status: 200 }; -``` - -### LLM Proxy with Streaming - -```js -// hooks/ai/post.before -var req = request(); -var body = json.parse(req.body()); - -var stream = http.fetchStream("https://api.openai.com/v1/chat/completions", { - method: "POST", - headers: { - "Content-Type": "application/json", - "Authorization": "Bearer " + context.config.server().llmApiKeys.openai - }, - body: context.json.stringify({ - model: "gpt-4", - messages: body.messages, - stream: true - }), - timeout: 0 -}); - -context.response.header("Content-Type", "text/event-stream"); -context.response.writeHeader(200); - -var line; -while ((line = stream.body.read()) !== null) { - if (line.indexOf("data: ") === 0) { - context.response.write(line + "\n\n"); - context.response.flush(); - } -} -stream.body.close(); - -throw { status: 200 }; -``` diff --git a/src/content/docs/server/07-realtime.md b/src/content/docs/server/07-realtime.md deleted file mode 100644 index baf3d52..0000000 --- a/src/content/docs/server/07-realtime.md +++ /dev/null @@ -1,190 +0,0 @@ ---- -title: 'Realtime Channels (SSE Pub/Sub)' ---- - -# Realtime Channels (SSE Pub/Sub) - -## Overview - -Tibi-Server includes an in-memory Pub/Sub hub for real-time communication via **Server-Sent Events (SSE)**. Each project has its own isolated `Hub` instance. Channels are created on-demand when the first subscriber connects, and automatically garbage-collected when the last subscriber leaves. - -## Architecture - -``` -Client A ─── SSE ──→ [Subscription] ──→ [Channel "chat"] ←── [Subscription] ←── SSE ──── Client B - │ - [Ring Buffer] - │ -Client C ─── SSE ──→ [Subscription] ──→ [Channel "chat"] - │ - Hook/Job ────────→ hub.Send("chat", data) -``` - -### Components - -| Component | Description | -|-----------|-------------| -| **Hub** | Per-project container for all channels. Thread-safe via `sync.RWMutex`. | -| **Channel** | Named pub/sub channel with a fixed-size ring buffer. Created on first subscribe. | -| **Subscription** | Per-client connection to a channel. Each gets independent read position. | -| **Message** | Wrapper for data with `CreatedAt` timestamp. | - -## Data Structures - -### Message - -```go -type Message struct { - Data interface{} - CreatedAt time.Time -} -``` - -### SubscribeOptions - -```go -type SubscribeOptions struct { - BufferSize int // Ring buffer capacity (default: 256, set by first subscriber) - OnFull OnFullStrategy // What to do when subscriber is slow - MessageTTL time.Duration // Skip messages older than this (0 = no TTL) - LastN int // Replay last N messages on connect (0 = none) - MaxAge time.Duration // Replay messages newer than this (0 = none) -} -``` - -### OnFullStrategy - -| Strategy | Value | Behavior | -|----------|-------|----------| -| `DropOldest` | `0` (default) | Slow subscriber's read pointer jumps forward (may miss messages) | -| `DropNewest` | `1` | New message is silently discarded for this subscriber | - -## Ring Buffer - -The channel uses a fixed-size ring buffer for message storage: - -``` -Buffer size: 8 -Write pointer: writeSeq (monotonically increasing) - -Position = writeSeq % size - -[0] [1] [2] [3] [4] [5] [6] [7] - ↑ ↑ - oldest writeSeq -``` - -- **First subscriber** determines the buffer size (subsequent subscribers use the existing size) -- Messages are deep-copied to each subscriber (independent mutation) -- `oldest = max(0, writeSeq - size)` — positions older than this have been overwritten - -## Replay Logic - -When a new subscriber connects with `LastN` or `MaxAge`: - -1. `readSeq` starts at `writeSeq` (only new messages by default) -2. If `LastN > 0`: candidate = `writeSeq - LastN`, clamped to `oldest` -3. If `MaxAge > 0`: backward scan from `writeSeq` until a message older than `MaxAge` is found -4. The lower value of both is used, but never below `oldest` - -This means subscribers immediately receive matching historical messages. - -## Message TTL - -If `MessageTTL` is set on a subscription, messages older than the TTL are silently skipped during `tryRead()`. This prevents slow subscribers from receiving stale data after reconnection. - -## Usage from Hooks - -### Subscribe (Blocking SSE Endpoint) - -```js -// hooks/events/get.before -context.response.header("Content-Type", "text/event-stream"); -context.response.header("Cache-Control", "no-cache"); -context.response.header("Connection", "keep-alive"); -context.response.writeHeader(200); - -var sub = channel.subscribe("notifications", { - bufferSize: 100, // Ring buffer size - onFull: "drop-newest", // "drop-oldest" (default) or "drop-newest" - messageTTL: 60000, // 60 seconds - lastN: 10, // Replay last 10 messages - maxAge: 30000 // Only replay messages from last 30 seconds -}); - -var msg; -while ((msg = sub.receive()) !== null) { - context.response.writeSSE("message", context.json.stringify(msg)); -} -sub.close(); -throw { status: 200 }; -``` - -### Send (From Any Hook or Job) - -```js -// hooks/posts/post.after -context.channel.send("notifications", { - type: "new_post", - id: context.data.id, - title: context.data.title, - timestamp: new Date().toISOString() -}); -``` - -## Receive Behavior - -| Return Value | Meaning | -|-------------|---------| -| `(data, nil)` | Message received (deep-copied) | -| `null` | Client disconnected (request context cancelled) | -| Error | Subscription, channel, or hub was closed | - -`receive()` blocks until: -- A new message arrives -- The client disconnects -- The subscription/channel/hub is closed - -## Automatic Cleanup - -- `channel.subscribe()` calls `DisableTimeout()` — the hook timeout is disabled -- A cleanup function is registered via `RegisterCleanup()` — the subscription is closed when the hook ends -- When the last subscriber leaves a channel, the channel is removed from the hub - -## Client-Side SSE - -```javascript -const eventSource = new EventSource('/api/v1/myproject/events'); - -eventSource.addEventListener('message', function(e) { - const data = JSON.parse(e.data); - console.log('Received:', data); -}); - -eventSource.onerror = function(e) { - console.log('Connection lost, reconnecting...'); -}; -``` - -## SSE Wire Format - -Messages sent via `response.writeSSE()` follow the standard SSE format: - -``` -event: message -data: {"type":"new_post","id":"abc123"} - -event: message -data: {"type":"update","id":"def456"} - -``` - -Each message is terminated by `\n\n` and flushed immediately. - -## Limitations - -- **In-memory only** — messages are lost on server restart -- **Single-instance** — no cross-server channel synchronization -- **No persistence** — no message durability guarantees -- **Buffer overflow** — slow subscribers may lose messages (ring buffer wrap-around) -- **No authentication on channels** — access is controlled by the collection's permission system that the hook runs in diff --git a/src/content/docs/server/08-file-upload-images.md b/src/content/docs/server/08-file-upload-images.md deleted file mode 100644 index 90d5abf..0000000 --- a/src/content/docs/server/08-file-upload-images.md +++ /dev/null @@ -1,292 +0,0 @@ ---- -title: 'File Uploads & Image Processing' ---- - -# File Uploads & Image Processing - -## File Upload Mechanism - -Files are uploaded as **Base64 data URIs** embedded in JSON document fields of type `file` or `file[]`. - -### Upload Format - -```json -{ - "avatar": "data:image/jpeg;base64,/9j/4AAQSkZJRg...", - "documents": [ - "data:application/pdf;base64,JVBERi0xLjQ...", - "data:text/plain;base64,SGVsbG8gV29ybGQ=" - ] -} -``` - -### How It Works - -1. **POST/PUT request** contains a `file` field with a Base64 data URI -2. The server detects the `data:` prefix in `handleFileInJSON()` -3. The Base64 content is decoded -4. A file is written to: `{uploadPath}/{documentId}/{fieldName}/{filename}` -5. The data URI in the document is replaced with the relative file path -6. The stored document contains only the path reference, not the binary data - -### File Path Structure - -``` -{uploadPath}/ - {documentId}/ - {fieldName}/ - {originalFilename} -``` - -- `uploadPath` is configured per collection (default: derived from project config) -- File permissions: `0755` -- Operations run under the project's configured UID (suexec) - -### Retrieving Files - -``` -GET /api/v1/{namespace}/{collection}/{id}/{field}/{filename} -``` - -Example: -``` -GET /api/v1/blog/posts/60a1b2c3d4e5f6/avatar/photo.jpg -``` - -### File Deletion - -When a document is deleted, all associated files in `{uploadPath}/{documentId}/` are removed automatically via `os.RemoveAll`. - -### Size Limits - -**Request Body Size** (Middleware — checked before any processing): -- Server-level: `api.maxBodySize` (default: 100MB) — absolute hard cap -- Project-level: `maxBodySize` in project config — default for all collections -- Collection-level: `maxBodySize` in collection config — overrides project (can increase, but never above server) -- Returns **413** if exceeded - -**Per-Field File Size** (checked during Base64 decode): -```yaml -fields: - - name: avatar - type: file - validator: - maxFileSize: "500KB" # Supports B, KB, MB, GB, TB -``` -- Uses `io.LimitReader` for memory-efficient streaming check -- Returns **400** if exceeded (file is deleted from disk) -- Works for both `file` and `file[]` fields -- Enforced on POST and PUT - -**Per-Field Image Dimensions** (checked from the decoded image header): -```yaml -fields: - - name: heroImage - type: file - validator: - accept: ["image/jpeg", "image/png", "image/webp"] - image: - minWidth: 1200 - maxWidth: 3200 - minHeight: 600 - maxHeight: 2400 -``` -- Works for both `file` and `file[]` fields -- Applies to newly uploaded Data-URI files on POST and PUT -- Validates dimensions before the file is kept on disk -- Currently supported for dimension checks: PNG, JPEG, GIF, BMP, TIFF, WebP, and SVG -- `allowNonImages: true` lets a mixed-media field accept non-image uploads unchanged; image uploads on the same field are still checked against `minWidth`, `maxWidth`, `minHeight`, and `maxHeight` -- `allowUnknownDimensions: true` turns the validator into a best-effort check for `image/*`: uploads with unknown dimensions are accepted and min/max checks are skipped for that file -- HEIF and HEIC are currently detected as image MIME types, but are not yet supported by strict dimension checks; with `allowUnknownDimensions: true` they can still pass if the field accepts them - ---- - -## Image Filters - -Image filters allow on-the-fly image processing when serving files. Filters are configured per collection and applied via query parameter. - -### Configuration - -```yaml -collections: - - name: posts - imageFilter: - thumbnail: - - width: 200 - height: 200 - fit: true - quality: 80 - outputType: webp - hero: - - width: 1200 - height: 600 - fill: true - anchor: center - quality: 85 -``` - -### Usage - -``` -GET /api/v1/blog/posts/{id}/image/photo.jpg?filter=thumbnail -``` - -### Caching - -Filtered images are cached on disk as: -``` -{originalPath}.{filterName}.{extension} -``` - -Example: `photo.jpg` with filter `thumbnail` → `photo.thumbnail.webp` - -Use `?no-cache=1` to force regeneration. - ---- - -## FilterParams Reference - -Each filter is an array of processing steps. All parameters are optional. - -### Resize Parameters - -| Parameter | Type | Description | -|-----------|------|-------------| -| `width` | int | Target width in pixels | -| `height` | int | Target height in pixels | -| `fit` | bool | Resize to fit within dimensions (preserves aspect ratio, no crop) | -| `fill` | bool | Resize to fill dimensions (crops to fit, uses `anchor`) | -| `resampling` | string | Resampling algorithm (see below) | -| `anchor` | string | Crop anchor point for `fill` mode (see below) | - -If neither `fit` nor `fill` is set, `imaging.Resize` is used (stretches to exact dimensions). - -### Effect Parameters - -| Parameter | Type | Range | Description | -|-----------|------|-------|-------------| -| `brightness` | float64 | -100 to 100 | Adjust brightness | -| `saturation` | float64 | -100 to 100 | Adjust saturation | -| `contrast` | float64 | -100 to 100 | Adjust contrast | -| `gamma` | float64 | > 0 | Gamma correction | -| `blur` | float64 | ≥ 0 | Gaussian blur sigma | -| `sharpen` | float64 | ≥ 0 | Unsharp mask sigma | -| `invert` | bool | — | Invert colors | -| `grayscale` | bool | — | Convert to grayscale | - -### Output Parameters - -| Parameter | Type | Description | -|-----------|------|-------------| -| `quality` | int | JPEG/WebP quality (1-100) | -| `lossless` | bool | WebP lossless mode | -| `outputType` | string | Output format: `jpeg`, `png`, `webp`, `gif`, `bmp`, `tiff` | -| `skipLargerDimension` | bool | Skip processing if source is smaller than target | -| `skipLargerFilesize` | bool | Keep original if processed file would be larger | - -### Resampling Algorithms - -| Value | Algorithm | Quality/Speed | -|-------|-----------|---------------| -| `nearestNeighbor` | Nearest Neighbor | Fastest, lowest quality | -| `box` | Box | Fast, low quality | -| `linear` | Bilinear | Good balance | -| `hermite` | Hermite | Good quality | -| `mitchellNetravali` | Mitchell-Netravali | High quality | -| `catmullRom` | Catmull-Rom | High quality | -| `bSpline` | B-Spline | High quality, smooth | -| `gaussian` | Gaussian | High quality | -| `lanczos` | Lanczos | Highest quality, slowest | -| `bartlett` | Bartlett | Good quality | -| `hann` | Hann | Good quality | -| `hamming` | Hamming | Good quality | -| `blackman` | Blackman | High quality | -| `welch` | Welch | Good quality | -| `cosine` | Cosine | Good quality | - -### Anchor Positions - -Used with `fill: true` to determine crop position: - -| Value | Position | -|-------|----------| -| `center` | Center (default) | -| `topLeft` | Top-left | -| `top` | Top-center | -| `topRight` | Top-right | -| `left` | Center-left | -| `right` | Center-right | -| `bottomLeft` | Bottom-left | -| `bottom` | Bottom-center | -| `bottomRight` | Bottom-right | - ---- - -## Supported Image Formats - -### Input Formats - -| Format | Library | -|--------|---------| -| JPEG | `imaging.Open` | -| PNG | `imaging.Open` | -| GIF | `imaging.Open` | -| BMP | `imaging.Open` | -| TIFF | `imaging.Open` | -| WebP | `golang.org/x/image/webp` decoder | - -### Output Formats - -| Format | Library | -|--------|---------| -| JPEG | `imaging` | -| PNG | `imaging` | -| WebP | `chai2010/webp` encoder | - -Auto-detect: If `outputType` is not set, format is determined from the target file extension. - ---- - -## Processing Pipeline - -When multiple filter steps are defined, they are applied **sequentially**: - -```yaml -imageFilter: - optimized: - - width: 1200 # Step 1: Resize - height: 800 - fit: true - sharpen: 0.5 - - quality: 80 # Step 2: Compress - outputType: webp -``` - -Per step: -1. Skip check (if `skipLargerDimension` and source is smaller) -2. Resize (`fit`, `fill`, or plain resize) -3. Apply effects (blur, brightness, contrast, gamma, saturation, sharpen, invert, grayscale) -4. Encode to output format with quality settings -5. Skip check (if `skipLargerFilesize` and result is larger) - -If the final file would be larger than the source and `skipLargerFilesize` is set, the original file is copied to the target path instead. - ---- - -## Image Processing in Hooks - -The `image.filter()` function provides the same processing capabilities in hooks: - -```js -context.image.filter("/path/to/source.jpg", "/path/to/output.webp", [ - { - width: 800, - height: 600, - fit: true, - quality: 85, - outputType: "webp" - } -]); -``` - -See [06-hooks.md](06-hooks.md#image--image-processing) for full API reference. diff --git a/src/content/docs/server/09-llm-integration.md b/src/content/docs/server/09-llm-integration.md deleted file mode 100644 index 5c6ff6e..0000000 --- a/src/content/docs/server/09-llm-integration.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -title: 'LLM Integration' ---- - -# LLM Integration - -## Overview - -Tibi-Server provides a built-in LLM proxy that supports OpenAI-compatible APIs with streaming, token budgets, and usage logging. Users can interact with LLMs through a unified API that handles provider resolution, model validation, and quota management. - -## Configuration - -### Server Config - -```yaml -llm: - providers: - - name: openai - type: openai # "openai" or "openai-compatible" - apiKey: sk-... # API key - baseURL: https://api.openai.com/v1 # Optional, defaults to OpenAI - defaultModel: gpt-4 # Fallback model - models: # Allowed models (whitelist) - - gpt-4 - - gpt-4-turbo - - gpt-3.5-turbo - maxTokensPerRequest: 4096 # Token limit per request - - name: local-llama - type: openai-compatible - apiKey: "" - baseURL: http://localhost:11434/v1 - defaultModel: llama3 - models: - - llama3 - - codellama -``` - -### API Key via Environment Variable - -```yaml -llm: - providers: - - name: openai - type: openai - apiKey: ${OPENAI_API_KEY} # Resolved from environment -``` - -Convention: `LLM__APIKEY` (e.g., `LLM_OPENAI_APIKEY`). - -## API Endpoints - -### Chat Completion - -```text -POST /api/v1/llm/chat -X-Auth-Token: -Content-Type: application/json -``` - -**Request:** - -```json -{ - "provider": "openai", - "model": "gpt-4", - "messages": [ - { "role": "system", "content": "You are a helpful assistant." }, - { "role": "user", "content": "Hello!" } - ], - "stream": false, - "temperature": 0.7, - "maxTokens": 1000 -} -``` - -**Response (non-streaming):** - -```json -{ - "id": "chatcmpl-abc123", - "model": "gpt-4", - "choices": [ - { - "index": 0, - "message": { "role": "assistant", "content": "Hello! How can I help?" }, - "finishReason": "stop" - } - ], - "usage": { - "promptTokens": 25, - "completionTokens": 10, - "totalTokens": 35 - } -} -``` - -### Streaming - -```json -{ - "provider": "openai", - "model": "gpt-4", - "messages": [{ "role": "user", "content": "Tell me a story." }], - "stream": true -} -``` - -**Response:** SSE stream with `Content-Type: text/event-stream` - -```text -data: {"id":"chatcmpl-abc","model":"gpt-4","choices":[{"index":0,"delta":{"role":"assistant","content":"Once"},"finishReason":""}]} - -data: {"id":"chatcmpl-abc","model":"gpt-4","choices":[{"index":0,"delta":{"content":" upon"},"finishReason":""}]} - -data: {"id":"chatcmpl-abc","model":"gpt-4","choices":[{"index":0,"delta":{"content":" a"},"finishReason":""}]} - -data: [DONE] -``` - -### List Providers - -```text -GET /api/v1/llm/providers -X-Auth-Token: -``` - -**Response:** - -```json -[ - { - "name": "openai", - "type": "openai", - "defaultModel": "gpt-4", - "models": ["gpt-4", "gpt-4-turbo", "gpt-3.5-turbo"] - }, - { - "name": "local-llama", - "type": "openai-compatible", - "defaultModel": "llama3", - "models": ["llama3", "codellama"] - } -] -``` - -## Model Resolution - -1. If `model` is provided in the request → validated against provider's `models` whitelist -2. If `model` is empty → provider's `defaultModel` is used -3. If the model is not in the whitelist → `400 Bad Request` - -## Token Budget System - -Administrators can assign per-provider token budgets to users: - -```json -{ - "username": "editor1", - "llmBudgets": [ - { - "provider": "openai", - "tokenBudget": 100000, - "tokensUsed": 45230, - "periodStart": "2024-01-01T00:00:00Z", - "periodEnd": "2024-02-01T00:00:00Z" - } - ] -} -``` - -### LLMBudget Structure - -| Field | Type | Description | -| ------------- | ------ | ---------------------------------- | -| `provider` | string | Provider name | -| `tokenBudget` | int | Maximum tokens allowed in period | -| `tokensUsed` | int | Tokens consumed so far | -| `periodStart` | time | Budget period start | -| `periodEnd` | time | Budget period end | - -### Budget Check Flow - -1. Before each LLM request, **user budgets** are read from the database -2. `CheckBudget()` verifies: - - Budget period has not expired (`PeriodEnd` > now) - - Tokens used < token budget -3. If budget exceeded → `429 Too Many Requests` -4. If the user has a `primaryOrgId`, the **org's budget** is checked as well (soft check — only fails if the org has a budget for this provider and it's exhausted) -5. After completion, `RecordUsage()` increments `tokensUsed` on both the user document and the org document (if applicable) - -### Org-Level Budget Cascade - -Orgs can have their own `llmBudgets` (see [Orgs & Teams](18-orgs-teams.md#llm-budget-cascade)). When a user with a `primaryOrgId` makes an LLM request: - -- **User budget** is checked first (hard check — must exist and have quota) -- **Org budget** is checked second via `user.primaryOrgId` (soft check — only enforced if configured) -- Usage is deducted from both budgets -- The `llm_usage` log entry includes both `userId` and `orgId` - -This allows multiple users to share an org-wide token pool while each maintaining individual quotas. - -### Token Estimation - -When the upstream provider does not return usage information, tokens are estimated: - -```go -// ~4 characters per token -func EstimateTokens(text string) int { - return len(text) / 4 -} - -// +4 tokens per message for framing overhead -func EstimateUsage(messages []ChatMessage, completionContent string) UsageInfo -``` - -## Usage Logging - -Every LLM request is logged to the `llm_usage` collection in the system database: - -```json -{ - "id": "...", - "userId": "60a1b2c3d4...", - "orgId": "507f1f77bcf86cd799439011", - "provider": "openai", - "model": "gpt-4", - "promptTokens": 25, - "completionTokens": 10, - "totalTokens": 35, - "timestamp": "2024-01-15T10:30:00Z", - "insertTime": "2024-01-15T10:30:00Z" -} -``` - -The `orgId` field is populated from the user's `primaryOrgId` when present, linking usage to the billed org. - -## Authentication - -All LLM endpoints require a valid JWT token (`X-Auth-Token` header). Both admin and regular users can use LLM endpoints, but only admins can modify user `llmBudgets`. - -## Provider Implementation - -Tibi-Server uses the **OpenAI API format** as its universal protocol. All providers (including `openai-compatible`) implement the same interface: - -```go -type Provider interface { - Name() string - ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error) - ChatCompletionStream(ctx context.Context, req ChatRequest) (*ChatStream, error) -} -``` - -This means any OpenAI-compatible API (Ollama, vLLM, Azure OpenAI, LiteLLM, etc.) can be used by setting `type: openai-compatible` with the appropriate `baseURL`. - -## Streaming Implementation Details - -- Streaming uses `bufio.Scanner` for SSE parsing -- The `stream_options.include_usage = true` flag is set for OpenAI to get usage info in the final chunk -- `ChatStream.Read()` returns `io.EOF` when `data: [DONE]` is received -- Usage information from the final chunk is captured via `GetUsage()` diff --git a/src/content/docs/server/10-audit.md b/src/content/docs/server/10-audit.md deleted file mode 100644 index 4c5edc4..0000000 --- a/src/content/docs/server/10-audit.md +++ /dev/null @@ -1,277 +0,0 @@ ---- -title: 'Audit Log' ---- - -# Audit Log - -## Overview - -Tibi-Server provides an audit logging system that records data changes (create, update, delete) with full snapshots. Audit logs are stored in the system database and queryable via API with role-based visibility. - -## Configuration - -### Server-Level Configuration - -```yaml -audit: - enabled: true - defaultTTL: "720h" # Go duration (30 days). "0" = keep forever. -``` - -### Project-Level Configuration - -```yaml -# in project config.yml (opt-in, default false for gradual migration) -audit: - enabled: true -``` - -> **Migration:** Audit is only active when **both** `server.audit.enabled` **and** `project.audit.enabled` are `true`. Existing projects without `audit.enabled` remain dark even when the server flag is enabled - add `audit.enabled: true` per project you want to migrate. Project level only supports `enabled` (no `actions`). - -### Collection-Level Configuration - -```yaml -collections: - - name: posts - audit: - enabled: true - actions: - - create - - update - - delete -``` - -**Default audited actions:** `create`, `update`, `delete` - -**Available collection-level actions:** `create`, `update`, `delete`, `get` - -**Available action-level actions (for `actions/*.yml`):** `post`, `get` (default `[post, get]`, `enabled` defaults to `true` if omitted) - -> **Note:** `db.deleteMany` and `db.updateMany` are audited as `delete` and `update` respectively — all covered by the default actions. -> -> **Note:** Bulk API operations (`POST /collection` with array body, `PUT /collection?filter=...` and `DELETE /collection?filter=...`) are audited as `bulkCreate`, `bulkUpdate` and `bulkDelete` with the filter/count logged. -> -> **Note:** `login`, `reload`, and `shutdown` are system-level actions that are always logged when `audit.enabled` is `true`. They cannot be controlled via collection-level `audit.actions`. - -### ShouldAudit Logic - -An action is audited if: - -1. Server-level `audit.enabled` is `true` -2. **Project-level `audit.enabled` is `true`** (opt-in, `nil`/`false` = off - `internal/models/auditlog.go:IsProjectAuditEnabled`) -3. Collection/Action-level `audit.enabled` is `true` (or not explicitly `false`) -3. The action is in the collection's `actions` list (or defaults are used) - -## AuditLog Model - -```json -{ - "id": "...", - "insertTime": "2024-01-15T10:30:00Z", - "userId": "60a1b2c3d4...", - "username": "admin", - "userRole": 0, - "projectNamespace": "myblog", - "projectName": "My Blog", - "collection": "posts", - "action": "update", - "documentId": "60a1b2c3d4e5f6...", - "source": { - "type": "controller", - "collection": "", - "method": "PUT", - "step": "", - "file": "", - "line": 0 - }, - "snapshot": { - "title": "Old Title", - "body": "Old content..." - }, - "changes": { - "title": "New Title" - }, - "filter": null, - "count": null, - "ip": "192.168.1.1", - "authMethod": "jwt", - "tokenLabel": "", - "tokenPrefix": "", - "channel": "mcp" -} -``` - -### Authentication Context - -Each audit entry includes authentication metadata: - -| Field | Description | Values | -|-------|-------------|--------| -| `authMethod` | How the request was authenticated | `jwt`, `admin-token`, or empty | -| `tokenLabel` | Human-readable label from admin token config | e.g. `ci-deploy`, `monitoring` | -| `tokenPrefix` | First 8 characters of the admin token | e.g. `agent-to` | -| `userId` | User ID (only for JWT auth) | MongoDB ObjectId | -| `username` | Username (only for JWT auth) | e.g. `admin` | -| `ip` | Client IP address | e.g. `192.168.1.1` | - -For JWT-authenticated requests, `userId`/`username`/`userRole` are populated and token fields are empty. -For admin token requests, `authMethod`/`tokenLabel`/`tokenPrefix` are populated and user fields are empty. -For PAT-authenticated requests (incl. MCP), `authMethod` is `access-token` with the token's name/prefix. - -### Channel (REST vs. MCP) - -| Field | Description | Values | -|-------|-------------|--------| -| `channel` | Through which interface the action came in | `mcp` for MCP tool calls, empty for REST (and older entries) | - -Filter MCP-originated entries with `?filter={"channel":"mcp"}`. The concrete -tool follows from `collection` + `action` (`documentId` set = single, `filter` -set = bulk). See [MCP-Server](23-mcp.md). - -### Source Types - -| Type | Description | -| ------------ | ------------------------------------------ | -| `controller` | Direct API request (CRUD endpoint) | -| `hook` | Database operation from a hook script | -| `system` | Internal system operation | -| `job` | Database operation from a scheduled job | - -For hook sources, additional fields are populated: - -- `collection` — the hook's parent collection -- `method` — the HTTP method that triggered the hook -- `step` — hook step (e.g. `read`, `return`, `bind`, `validate`, `create`, `update`, `delete`) -- `file` — JavaScript source file -- `line` — line number in the JS file (from goja callstack) - -## API - -### Query Audit Logs - -```text -GET /api/v1/audit -X-Auth-Token: -``` - -**Query Parameters:** - -| Parameter | Type | Default | Description | -| --------- | ----------- | -------------- | -------------------------------------------------------------------- | -| `filter` | JSON string | — | MongoDB-style filter (validated against AuditLog fields) | -| `limit` | int | 50 | Max results (capped at `maxLimit`, default max: 10,000) | -| `offset` | int | 0 | Pagination offset | -| `sort` | string | `-insertTime` | Sort field(s), prefix `-` for descending | -| `count` | string | — | `"1"` or `"true"` to include total count in `X-Results-Count` header | - -**Configurable Limits:** - -The audit endpoint enforces query result limits: - -- **Default limit:** 50 (configurable via `audit.defaultLimit` in server config) -- **Maximum limit:** 10,000 (configurable via `audit.maxLimit` in server config) -- If `defaultLimit` exceeds `maxLimit`, it is automatically capped to `maxLimit`. -- See [Configuration](02-configuration.md) for YAML and environment variable options. - -**Example:** - -```bash -curl -H "X-Auth-Token: $TOKEN" \ - 'http://localhost:8080/api/v1/audit?filter={"collection":"posts","action":"update"}&limit=20&sort=-insertTime' -``` - -### Role-Based Visibility - -**Admin (role=0):** - -- Sees all audit entries across all projects - -**User (role=1):** - -- Only sees entries for projects and collections where they have GET permission -- System-sourced entries (`source.type: "system"`) are hidden -- Snapshot data is filtered through the collection's projection rules (same include/exclude logic as MongoDB projections) - -### Sensitive Fields & audit.return Hooks - -Audit snapshots store the **full document** at the time of the operation. Fields that are normally stripped by `get.return` hooks (e.g. passwords, API keys) are still present in audit snapshots. - -To prevent sensitive data from being exposed through the audit endpoint, use the `audit.return` hook on the collection: - -```yaml -hooks: - audit: - return: - type: javascript - file: hooks/users/audit_return.js -``` - -```js -// hooks/users/audit_return.js -;(function () { - var results = context.results - if (results && results.length > 0) { - if (results[0].snapshot) { - delete results[0].snapshot.password - delete results[0].snapshot.apiKey - } - if (results[0].changes) { - delete results[0].changes.password - delete results[0].changes.apiKey - } - } - return { results: results } -})() -``` - -Key properties: -- Runs for **all roles** including admin -- Affects existing snapshots retroactively (filtering happens at read-time) -- Each entry is processed individually (`context.one = true`, `context.results` has exactly one entry) -- Returning `{ results: [] }` suppresses the entry from the response -- Throwing an exception aborts the entire audit request - -## Database Indexes - -Four indexes are automatically created on the audit collection: - -1. **TTL Index:** On `insertTime` — auto-deletes entries after configured TTL -2. **Project + Time:** `{projectNamespace, insertTime}` -3. **User + Time:** `{userId, insertTime}` -4. **Document:** `{collection, documentId, insertTime}` - -## Write Behavior - -- Audit log writes are **asynchronous by default** (fire-and-forget goroutine with own database session) -- Synchronous mode available via `AuditWriteSync = true` (used in tests) -- Each write uses its own MongoDB session to avoid blocking the main request - -## What Gets Logged - -### Controller Actions - -| Action | Snapshot | Changes | Filter | -| -------- | ------------------------ | ---------------- | ------ | -| `create` | — | Created document | — | -| `update` | Document before update | Update data | — | -| `delete` | Document before deletion | — | — | - -### Hook Database Actions - -| Action | Snapshot | Changes | Filter | -| --------------------- | ------------------------ | ---------------- | ------------------- | -| `db.create` | — | Created document | — | -| `db.update` | Document before update | Update data | — | -| `db.delete` | Document before deletion | — | — | -| `db.deleteMany` | — | — | Filter used + count | -| `db.updateMany` | — | Update data | Filter used + count | - -### Bulk API Actions - -| Action | Snapshot | Changes | Filter | -| ------------ | -------- | ----------- | ------------------- | -| `bulkCreate` | — | — | Count + IDs | -| `bulkUpdate` | — | Update data | Filter used + count | -| `bulkDelete` | — | — | Filter used + count | - -> `db.deleteMany` and `db.updateMany` are audited under the actions `delete` and `update` respectively. Bulk API actions (`bulkCreate`, `bulkUpdate`, `bulkDelete`) are also audited as `create`, `update` and `delete`. diff --git a/src/content/docs/server/11-jobs.md b/src/content/docs/server/11-jobs.md deleted file mode 100644 index e7bca60..0000000 --- a/src/content/docs/server/11-jobs.md +++ /dev/null @@ -1,154 +0,0 @@ ---- -title: 'Jobs (Scheduled Tasks)' ---- - -# Jobs (Scheduled Tasks) - -## Overview - -Jobs are scheduled JavaScript programs that run on a cron schedule. They use the same goja runtime as hooks but execute independently of HTTP requests. Jobs are defined per project in the `config.yml`. - -## Configuration - -```yaml -jobs: - - cron: "0 */6 * * *" # Every 6 hours - type: javascript - file: jobs/cleanup.js - timeout: 30000 # Milliseconds (optional) - meta: # Arbitrary metadata (optional) - maxAge: 86400 - - cron: "0 8 * * *" # Every day at 8:00 - type: javascript - file: jobs/daily_report.js - timeout: 60000 -``` - -### JobConfig Fields - -| Field | Type | Required | Description | -| --------- | ------ | -------- | ------------------------------------------------------ | -| `cron` | string | yes | Cron expression (standard 5-field format) | -| `type` | string | yes | Job type (currently only `"javascript"`) | -| `file` | string | yes | Path to JavaScript file (relative to config directory) | -| `timeout` | int | no | Execution timeout in **milliseconds** | -| `meta` | object | no | Arbitrary metadata accessible via `config.project()` | - -### Cron Expression Format - -```text -┌───────────── minute (0-59) -│ ┌───────────── hour (0-23) -│ │ ┌───────────── day of month (1-31) -│ │ │ ┌───────────── month (1-12) -│ │ │ │ ┌───────────── day of week (0-6, 0=Sunday) -│ │ │ │ │ -* * * * * -``` - -Examples: - -- `*/5 * * * *` — Every 5 minutes -- `0 * * * *` — Every hour -- `0 0 * * *` — Daily at midnight -- `0 8 * * 1-5` — Weekdays at 8:00 -- `0 0 1 * *` — First of each month - -## Available Context - -Jobs have access to most hook context packages, with the following **exceptions**: - -| Package | Available in Jobs? | Reason | -| ------------------- | ------------------ | ------------------------ | -| `db.*` | Yes | | -| `http.*` | Yes | | -| `fs.*` | Yes | | -| `smtp.*` | Yes | | -| `jwt.*` | Yes | | -| `bcrypt.*` | Yes | | -| `crypto.*` | Yes | | -| `base64.*` | Yes | | -| `json.*` | Yes | | -| `xml.*` | Yes | | -| `tpl.*` | Yes | | -| `image.*` | Yes | | -| `pdf.*` | Yes | | -| `exec.*` | Yes | | -| `config.*` | Yes | | -| `debug.*` | Yes | | -| `charset.*` | Yes | | -| `channel.send` | Yes | | -| `cookie.*` | **No** | No HTTP request/response | -| `user.*` | **No** | No HTTP request | -| `response.*` | **No** | No HTTP response | -| `request` | **No** | No HTTP request | -| `channel.subscribe` | **No** | No HTTP response for SSE | - -## Job Examples - -### Cleanup Old Documents - -```js -// jobs/cleanup.js -var threshold = new Date(); -threshold.setDate(threshold.getDate() - 30); - -var result = db.deleteMany("temp_data", { - filter: { - createdAt: { "$lt": { "$date": threshold.toISOString() } } - } -}); - -context.debug.dump("Cleaned up " + result.removed + " documents"); -``` - -### Daily Report Email - -```js -// jobs/daily_report.js -var yesterday = new Date(); -yesterday.setDate(yesterday.getDate() - 1); - -var newPosts = db.count("posts", { - filter: { - insertTime: { "$gte": { "$date": yesterday.toISOString() } } - } -}); - -var newUsers = db.count("_users", { - filter: { - insertTime: { "$gte": { "$date": yesterday.toISOString() } } - } -}); - -var html = tpl.execute( - context.fs.readFile(context.fs.configDir() + "/templates/daily_report.html"), - { newPosts: newPosts, newUsers: newUsers, date: yesterday.toISOString() } -); - -context.smtp.sendMail({ - to: "admin@example.com", - from: "reports@example.com", - subject: "Daily Report - " + yesterday.toISOString().split("T")[0], - html: html -}); -``` - -### Notify via Realtime Channel - -```js -// jobs/notify.js -context.channel.send("system", { - type: "scheduled_maintenance", - message: "Scheduled maintenance in 30 minutes", - timestamp: new Date().toISOString() -}); -``` - -## Execution - -- Jobs run in a `JOB_CONTEXT` (FunctionContextType = 2) -- Each job execution creates a fresh goja runtime -- Jobs execute under the project's configured UID (suexec) -- Job errors are logged server-side and reported to Sentry -- Audit log entries from job database operations have `source.type: "job"` diff --git a/src/content/docs/server/12-deployment.md b/src/content/docs/server/12-deployment.md deleted file mode 100644 index 8097c86..0000000 --- a/src/content/docs/server/12-deployment.md +++ /dev/null @@ -1,152 +0,0 @@ ---- -title: 'Deployment' ---- - -# Deployment - -## System Requirements - -| Component | Requirement | -|-----------|-------------| -| **Go** | 1.23+ (toolchain 1.24) | -| **MongoDB** | 4.4+ (with replica set for transactions) | -| **OS** | Linux (required for suexec UID isolation) | -| **wkhtmltopdf** | Required for PDF generation (optional) | - -## Docker Deployment - -### Production Dockerfile - -```dockerfile -# Multi-stage build -FROM golang:1.24 AS builder -WORKDIR /app -COPY . . -RUN go build -o server ./cmd/server - -FROM debian:bookworm-slim -RUN apt-get update && apt-get install -y ca-certificates wkhtmltopdf && rm -rf /var/lib/apt/lists/* -COPY --from=builder /app/server /usr/local/bin/server -CMD ["server"] -``` - -### Docker Compose - -```yaml -version: '3.8' -services: - tibi: - build: . - ports: - - "8080:8080" - volumes: - - ./config.yml:/app/config.yml - - ./projects:/app/projects - environment: - - API_MONGOURI=mongodb://mongo:27017 - - API_JWTSECRET=your-jwt-secret-here - depends_on: - - mongo - - mongo: - image: mongo:7 - command: ["--replSet", "rs0"] - volumes: - - mongo-data:/data/db - ports: - - "27017:27017" - -volumes: - mongo-data: -``` - -### Development with Air (Hot Reload) - -A `Dockerfile.air` is provided for development with [Air](https://github.com/cosmtrek/air) hot-reloading: - -```bash -docker compose -f docker-compose-local.yml up -``` - -## Configuration File - -See [02-configuration.md](02-configuration.md) for the complete configuration reference. - -Minimal `config.yml`: - -```yaml -api: - port: 8080 - mongoUri: mongodb://localhost:27017 - jwtSecret: change-me-to-a-long-random-string - dbPrefix: tibi - projectsDir: ./projects -``` - -## UID/GID Isolation (suexec) - -When running as **root**, tibi-server uses thread-level UID/GID isolation for multi-tenant security: - -### How It Works - -1. The main goroutine locks its OS thread at startup (`runtime.LockOSThread()`) -2. For each unique UID/GID pair, a dedicated goroutine is spawned that: - - Locks its OS thread - - Changes thread credentials via `SYS_SETGID` / `SYS_SETUID` syscalls - - Waits for jobs in an event loop -3. File operations and `exec.command` calls are dispatched to the appropriate UID thread -4. Thread pool scales up to `runtime.NumCPU()` per UID/GID pair - -### Thread Pool Behavior - -- New threads are spawned on 1-second send timeout (pool is busy) -- Maximum threads per UID/GID = `runtime.NumCPU()` -- Threads persist for the lifetime of the server - -### File-Based UID Detection - -`RunInFilesUID(filename)` determines the UID/GID from `os.Stat()` of the given file. This is used for project data directories — the filesystem owner determines the execution context. - -### When Not Root - -When running as a non-root user, suexec is not available. All file operations run under the server process's UID. - -## Graceful Shutdown - -The server supports graceful shutdown via: - -1. **OS Signal:** `SIGINT` or `SIGTERM` -2. **Admin API:** `POST /api/v1/admin/shutdown` with admin token (requires `server.shutdown` permission) - -Shutdown process: -1. HTTP server stops accepting new connections -2. In-flight requests are allowed to complete (with timeout) -3. All realtime hub instances are closed (wakes blocking subscriptions) -4. MongoDB connections are closed - -## Admin Reload - -``` -POST /api/v1/_/{project}/_/admin/reload -X-Admin-Token: your-admin-token -``` - -Reloads a specific project's configuration from disk without restarting the server. This: -- Re-reads all `config.yml` files from `projectsDir` -- Recompiles all hooks and jobs -- Rebuilds collection indexes -- Does **not** drop existing realtime connections (they continue on the old config until they reconnect) - -## Health Check - -``` -GET /api/v1/version -``` - -Returns server version information. No authentication required. Suitable for load balancer health checks. - -## Monitoring - -- **Prometheus Metrics:** `GET /metrics` — see [13-openapi-metrics.md](13-openapi-metrics.md) -- **Sentry Integration:** Configure `api.sentry` in `config.yml` for error tracking -- **Audit Logs:** Enable `audit.enabled: true` for data change tracking diff --git a/src/content/docs/server/13-openapi-metrics.md b/src/content/docs/server/13-openapi-metrics.md deleted file mode 100644 index 5e814c9..0000000 --- a/src/content/docs/server/13-openapi-metrics.md +++ /dev/null @@ -1,151 +0,0 @@ ---- -title: 'OpenAPI & Metrics' ---- - -# OpenAPI & Metrics - -## OpenAPI Specification - -Tibi-Server auto-generates an OpenAPI 3.0.0 specification for each project based on its collection configuration. - -### Endpoint - -``` -GET /api/v1/{namespace}/openapi -``` - -No authentication required. - -### Query Parameters - -| Parameter | Default | Description | -|-----------|---------|-------------| -| `lang` | `en` | Language for i18n fields (resolves `{en: "...", de: "..."}` maps) | - -### Generated Content - -The specification includes: - -1. **Info:** Project name/namespace as title, version `"0.0.0"` -2. **Paths:** Two paths per collection: - - `/{collection}` — List endpoint (GET) - - `/{collection}/{id}` — Single document endpoint (GET) -3. **Schemas:** Generated from collection field definitions (recursive for nested objects) -4. **Security Schemes:** API key schemes derived from token-based permissions - -### Collection Metadata for OpenAPI - -Collections can customize their OpenAPI representation via `meta.openapi`: - -```yaml -collections: - - name: posts - meta: - openapi: - disabled: false # Exclude from spec entirely - get: - disabled: false # Exclude list endpoint - summary: "List posts" - description: "Returns all published posts" - getOne: - disabled: false # Exclude single endpoint - summary: "Get post" - description: "Returns a single post by ID" -``` - -### Type Mapping - -| Collection Field Type | OpenAPI Type | -|----------------------|--------------| -| `string` | `string` | -| `int` | `integer` | -| `float` | `number` | -| `bool` | `boolean` | -| `date` | `string` (format: date-time) | -| `object` | `object` (with nested properties) | -| `file` | `object` with `{ src: string }` | -| `string[]` | `array` of `string` | -| (other arrays) | `array` of corresponding type | - -### i18n Support - -Field names, descriptions, and summaries can be i18n maps: - -```yaml -fields: - - name: - en: Title - de: Titel - key: title - type: string -``` - -The `lang` query parameter determines which language string is used in the generated spec. - ---- - -## Prometheus Metrics - -### Endpoint - -``` -GET /metrics -``` - -Registered on the root router (not under `/api/v1/`). Standard Prometheus text format. No authentication required. - -### Exposed Metrics - -#### `tibi_request_duration_seconds` (Summary) - -Tracks response time for all HTTP handlers. - -| Label | Description | Example | -|-------|-------------|---------| -| `method` | HTTP method | `GET`, `POST`, `PUT`, `DELETE` | -| `status` | HTTP status code | `200`, `404`, `500` | -| `base` | System collection name (non-project routes) | `user`, `project`, `login` | -| `ns` | Project namespace (project routes) | `myblog`, `myshop` | -| `col` | Collection name | `posts`, `products` | -| `id` | Document ID indicator | `1` (with ID) or `""` (list) | - -**Quantiles:** - -| Quantile | Error Margin | -|----------|-------------| -| p50 | ±0.05 | -| p75 | ±0.025 | -| p90 | ±0.01 | -| p99 | ±0.001 | - -### Example Output - -``` -# HELP tibi_request_duration_seconds Histogram of response time for handler in seconds -# TYPE tibi_request_duration_seconds summary -tibi_request_duration_seconds{method="GET",status="200",base="",ns="myblog",col="posts",id="",quantile="0.5"} 0.002 -tibi_request_duration_seconds{method="GET",status="200",base="",ns="myblog",col="posts",id="",quantile="0.9"} 0.005 -tibi_request_duration_seconds{method="GET",status="200",base="",ns="myblog",col="posts",id="",quantile="0.99"} 0.012 -tibi_request_duration_seconds_sum{method="GET",status="200",base="",ns="myblog",col="posts",id=""} 1.234 -tibi_request_duration_seconds_count{method="GET",status="200",base="",ns="myblog",col="posts",id=""} 500 -``` - -### Grafana Dashboard Example - -Query for average request duration per endpoint: - -```promql -rate(tibi_request_duration_seconds_sum[5m]) / rate(tibi_request_duration_seconds_count[5m]) -``` - -Query for p99 latency per collection: - -```promql -tibi_request_duration_seconds{quantile="0.99", ns="myblog"} -``` - -### Integration - -The metrics middleware (`measureResponseDuration`) is registered globally on all routes. It measures the duration of each request and records it with the appropriate labels. - -Standard Go runtime metrics are also available via the default Prometheus registry (`go_*` metrics). diff --git a/src/content/docs/server/14-security.md b/src/content/docs/server/14-security.md deleted file mode 100644 index 17ae98d..0000000 --- a/src/content/docs/server/14-security.md +++ /dev/null @@ -1,636 +0,0 @@ ---- -title: 'Security Audit — Tibi-Server' ---- - -# Security Audit — Tibi-Server - -> **Audit-Datum:** 27. Februar 2026 -> **Scope:** Vollständiges Security-Audit des tibi-server Quellcodes -> **Methode:** Statische Code-Analyse aller sicherheitsrelevanten Komponenten - ---- - -## Zusammenfassung - -| Schweregrad | Anzahl | Status | -|-------------|--------|--------| -| **KRITISCH** | 3 | Offen | -| **HOCH** | 6 | Offen | -| **MITTEL** | 7 | Offen | -| **NIEDRIG** | 5 | Offen / Teilweise gelöst | -| **Gelöst** | 1 | ✅ | - ---- - -## Security Architecture - -### Authentication Layer - -| Mechanism | Scope | Header/Cookie | -|-----------|-------|---------------| -| JWT (HMAC-SHA256) | User authentication | `X-Auth-Token` header | -| Refresh Token | Token renewal | `tibi-refresh-token` cookie (HttpOnly, SameSite=Strict) | -| Admin Tokens | Server-level CRUD, management & namespace collection access | `X-Admin-Token` header or `Authorization: Bearer` | -| Token Permissions | API key access | Configurable header/query parameter | - -### Authorization Layers - -1. **Role-Based:** Admin (role=0) bypasses all checks -2. **Project-Based:** Users assigned to projects via `project.users[]` -3. **Collection-Based:** Per-collection permission sets with method restrictions -4. **Pre-Filters:** Automatic query filters based on permission context -5. **Projection Restrictions:** Limit visible fields per permission level - -### Data Isolation - -- **Database:** Each project gets its own MongoDB database (`{prefix}_{namespace}`) -- **Filesystem:** suexec UID/GID isolation (Linux, requires root) -- **Realtime:** Each project has its own Hub instance - -### Input Validation - -- Collection field validation (type, required, allowZero, in, eval) -- MongoDB filter operator whitelist (dangerous operators blocked) -- File path sanitization (`CleanFilePath`) -- User input validation (username alphanumeric, password min 3 chars) - -### Bulk Operation Security - -Bulk operations (`POST` with JSON array body, `PUT`/`DELETE` without `:id`) require **two** permission checks: - -1. The standard method permission must be granted (`post`, `put` or `delete` with `allow: true`) -2. The `bulk` flag must be set to `true` in the object form of the permission - -Method permissions support two formats: - -- **Boolean shorthand:** `post: true` — allows single-document POST only (no bulk) -- **Object form:** `post: {allow: true, bulk: true}` — allows both single and bulk POST - -This applies equally to PUT and DELETE. Without explicit `bulk: true`, bulk endpoints return **403 Forbidden**. This ensures that existing projects are not automatically exposed to bulk operations when upgrading. - ---- - -## Findings - -### CRITICAL - -#### ~~C1. MD5 Password Hashing~~ ✅ BEHOBEN (2025-02-27) - -**Location:** `internal/models/login.go`, `internal/models/user.go` -**CVSS-Schätzung:** 9.1 (Kritisch) → **behoben** - -**Description:** User passwords wurden zuvor mit MD5 gehasht, was kryptographisch gebrochen ist. - -**Fix:** Passwort-Hashing wurde auf bcrypt (`golang.org/x/crypto/bcrypt`, DefaultCost) umgestellt mit abwärtskompatibler MD5-Migration: -- `user.go:ValidateData()` — neue Passwörter werden mit bcrypt gehasht -- `login.go:LoginCheck()` — Login prüft per `CheckPassword()` sowohl bcrypt als auch legacy-MD5 -- Transparente Migration: bei erfolgreichem Login mit MD5-Hash wird automatisch auf bcrypt re-gehasht und in der DB aktualisiert -- Neue Hilfsfunktionen: `IsBcryptHash()`, `CheckPassword()` (exportiert für Tests) -- Unit-Tests: `TestIsBcryptHash`, `TestCheckPassword` + alle bestehenden Tests grün - ---- - -#### C2. No Rate Limiting on Login Endpoint — ✅ BEHOBEN - -**Location:** `internal/ratelimit/ratelimit.go`, `internal/controller/ratelimit_middleware.go`, `internal/controller/setup.go` -**CVSS-Schätzung:** 8.8 (Hoch–Kritisch) - -**Description:** The `POST /api/v1/login` endpoint has no rate limiting or account lockout mechanism. Combined with MD5 password hashing and a minimum password length of only 3 characters, this makes the system highly vulnerable to brute-force attacks. - -**Impact:** An attacker can attempt unlimited login attempts without any delay, lockout, or CAPTCHA. With a 3-character minimum password, the search space is trivially small. - -**Lösung implementiert:** -- In-Memory Rate Limiter mit exponentiellem Backoff (1s → 2s → 4s → ... → 60s Cap) -- Duales Tracking: per IP und per Username (getrennt) -- Automatischer Reset nach 15 Minuten Inaktivität -- HTTP 429 + `Retry-After` Header bei Blockierung -- Konfigurierbar via `config.yml` (`rateLimit`-Sektion) und Environment-Variablen -- Standardmäßig aktiviert, kann via `rateLimit.enabled: false` deaktiviert werden -- Hintergrund-Cleanup-Goroutine für abgelaufene Einträge - ---- - -#### C3. Admin Token Comparison Vulnerable to Timing Attacks — ✅ BEHOBEN - -**Location:** `internal/config/admin_token.go`, `internal/controller/admin_reload.go` -**CVSS-Schätzung:** 7.5 (Hoch) → **behoben** - -**Description:** Admin tokens wurden mit Go's `==` Operator verglichen, was nicht constant-time ist und Timing-Angriffe ermöglicht. - -**Lösung implementiert:** -- Alle 3 Token-Vergleichsstellen auf `crypto/subtle.ConstantTimeCompare()` umgestellt: - - `config/admin_token.go:CheckAdminToken()` — Server-Admin-Token-Prüfung - - `config/admin_token.go:GetAdminTokenInfo()` — Admin-Token-Metadaten-Abfrage - - `controller/admin_reload.go:ProjectReload()` — Projekt-Admin-Token-Prüfung -- Constant-Time-Vergleich verhindert Rückschlüsse auf Token-Inhalte durch Zeitmessung - ---- - -### HIGH - -#### H1. Secrets in Plain Text Configuration - -**Location:** `config.yml`, `internal/config/server.go` -**CVSS-Schätzung:** 7.5 - -**Description:** JWT secrets, SMTP passwords, and LLM API keys are stored in plain text in the configuration file. The shipped `config.yml` contains a hardcoded JWT secret: - -```yaml -api: - jwtSecret: 237uiejakymxeou3zhakN # Hardcoded production secret in repo! -``` - -**Impact:** Anyone with read access to the repository or config file can forge JWT tokens, access SMTP credentials, and use LLM API keys. - -**Mitigation:** Environment variable substitution is available (`${ENV_VAR}` syntax). - -**Recommendation:** -- Replace all hardcoded secrets in `config.yml` with env-var references -- Warn on startup if `jwtSecret` does not use `${...}` syntax -- Add `.gitignore` entry for production configs -- Consider adding a vault integration - ---- - -#### H2. Token Authentication via Query Parameters — ℹ️ AKZEPTIERTES RISIKO - -**Location:** `internal/models/permission.go` -**CVSS-Schätzung:** 7.0 -**Status:** Akzeptiertes Risiko (bewusste Design-Entscheidung) + Timing-Attack-Fix - -**Description:** Token-based permissions check both headers and query parameters. Query parameters appear in server logs, browser history, referrer headers, and proxy logs, exposing authentication tokens. - -**Bewertung:** Die Query-Parameter-Unterstützung ist ein bewusstes Feature für Anwendungsfälle, in denen Header nicht gesetzt werden können (z.B. direkte URL-Aufrufe, Webhooks). Das Risiko ist dokumentiert und dem Betreiber bekannt. - -**Timing-Attack-Fix (2025-02-27):** Token-Vergleiche in `GetTokenPermissionSet()` wurden von `==` auf `subtle.ConstantTimeCompare()` umgestellt, um Timing-Angriffe auf Token-Werte zu verhindern. - -**Empfehlung für Betreiber:** -- Token-Werte über Environment-Variablen konfigurieren (`"x-api-key:${API_KEY}"`) -- Proxy/Loadbalancer so konfigurieren, dass Query-Parameter nicht geloggt werden -- Wo möglich, Header-basierte Token-Übergabe bevorzugen - ---- - -#### H3. Refresh Token Cookie Missing Secure Flag — ✅ BEHOBEN - -**Location:** `internal/models/login.go`, `internal/config/server.go` -**CVSS-Schätzung:** 7.0 → **behoben** - -**Description:** The refresh token cookie had `Secure: false` hardcoded. - -**Lösung implementiert:** -- Neuer Config-Eintrag `api.secureCookies` (bool, Default: `true`) -- Environment-Variable: `API_SECURE_COOKIES` -- Beide `SetCookie`-Aufrufe (Login + Logout) verwenden jetzt `config.Server.API.IsSecureCookies()` -- Default `true` → Cookies werden nur über HTTPS gesendet -- Für lokale Entwicklung (HTTP) kann `secureCookies: false` gesetzt werden - ---- - -#### H5. SSRF via http.fetch in Hooks - -**Location:** `internal/hook/context_http.go` -**CVSS-Schätzung:** 6.5 - -**Description:** The `http.fetch` and `http.fetchStream` functions in hooks allow requests to arbitrary URLs without restriction. A hook can fetch internal network resources (e.g. `http://169.254.169.254/` for cloud metadata, `http://localhost:27017/` for MongoDB). - -```go -request, err := http.NewRequest(method, url, reqBody) -// No URL validation — any scheme, any host, any port -``` - -**Impact:** Server-Side Request Forgery (SSRF) allowing access to internal services, cloud metadata endpoints, and potentially the MongoDB instance. - -**Recommendation:** -- Implement an optional URL whitelist/blocklist at config level -- Block requests to private IP ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `127.0.0.0/8`) by default -- Document SSRF risks for project administrators - ---- - -#### H6. Unrestricted Shell Command Execution in Hooks - -**Location:** `internal/hook/context_exec.go` -**CVSS-Schätzung:** 6.5 - -**Description:** The `exec.command` function executes any binary without path or name restrictions. While arguments are passed as separate `args[]` (safe from shell injection), the command name is controlled by hook code: - -```go -cmd := exec.CommandContext(timeoutCtx, command, cmdArgs...) -``` - -The `dir` option also allows setting arbitrary working directories: -```go -if dir, ok := options["dir"].(string); ok { - cmd.Dir = dir -} -``` - -**Mitigation:** Commands execute under project UID via suexec isolation. Default dir is project config directory. - -**Impact:** Arbitrary code execution on the host with project-level privileges. Without suexec (non-root deployment), commands run as the server process user. - -**Recommendation:** -- Implement a command whitelist at the project config level -- Restrict `dir` option through `CleanFilePath` validation -- Log all command executions for audit purposes - ---- - -### MEDIUM - -#### M1. Error Message Information Disclosure - -**Location:** `internal/utils/error.go`, `internal/dbapi/helper.go` -**CVSS-Schätzung:** 5.3 - -**Description:** By default (`config.response.error.message` not set), the raw error message is included in API responses: - -```go -errorInResponse := errorString -if config.Server.Response.Error.Message != "" { - errorInResponse = config.Server.Response.Error.Message -} -``` - -When `response.error.stack: true` is set, Go stack traces are exposed: -```go -if config.Server.Response.Error.Stack { - o["errorStack"] = errorStack -} -``` - -**Impact:** Detailed error messages can reveal internal paths, database structure, Go type names, and stack traces. - -**Recommendation:** -- Default to generic error messages in production -- Only enable detailed errors when explicitly configured for development -- Never expose stack traces in production - ---- - -#### M2. File Upload: No Size Limit or Type Validation - -**Location:** `internal/controller/collection.go:75-140` -**CVSS-Schätzung:** 5.3 - -**Description:** File uploads via base64-encoded data in JSON have no size limit or content type validation: - -```go -if i := strings.Index(srcPrefix, ","); i > 0 && - strings.HasPrefix(srcPrefix, "data:") && - strings.Index(srcPrefix, "base64") > 0 { // validate as data -``` - -The MIME type is extracted from the `data:` URL but not validated against an allowlist. Any file type can be uploaded. - -**Impact:** -- Denial of service via extremely large base64 payloads (decoded in memory) -- Storage exhaustion attacks -- Upload of executable files or malicious content - -**Recommendation:** -- Implement configurable maximum file size limits -- Add optional MIME type allowlists per collection field -- Consider limiting request body size at the middleware level (Gin's `MaxMultipartMemory`) - ---- - -#### M3. File Upload: Filename Not Sanitized - -**Location:** `internal/controller/collection.go:119-121` -**CVSS-Schätzung:** 5.3 - -**Description:** The original filename from the upload payload is used directly in the file path: - -```go -if filename, ok := fileObj["path"].(string); ok { - path += "/" + filename -} -``` - -While the `path` is within the upload directory tree, the `filename` value is not sanitized. A filename like `../../etc/cron.d/malicious` combined with upload path could potentially escape the intended directory. - -**Mitigation:** The upload path structure `uploadPath + "/" + id + "/" + field` provides some containment. The ID is server-generated. - -**Recommendation:** -- Sanitize filenames: strip directory separators, use `filepath.Base()`, restrict to alphanumeric + allowed extensions -- Apply `CleanFilePath` validation to constructed upload paths - ---- - -#### M4. ReDoS Risk via $regex Filter Operator - -**Location:** `internal/dbapi/filter_valid_functions.go:28` -**CVSS-Schätzung:** 5.3 - -**Description:** The `$regex` operator is allowed in user-provided query filters: - -```go -"$regex": true, -``` - -MongoDB's `$regex` uses PCRE, which is vulnerable to catastrophic backtracking with malicious patterns. An attacker can craft regex patterns that cause exponential CPU usage on the database server. - -**Impact:** Denial of service via CPU exhaustion on MongoDB. - -**Recommendation:** -- Set a timeout for MongoDB read operations -- Consider restricting `$regex` to authenticated users only -- Add regex complexity validation (max length, disallow nested quantifiers) -- Document ReDoS risks - ---- - -#### M5. Prometheus /metrics Endpoint Unauthenticated - -**Location:** `internal/controller/setup.go:135` -**CVSS-Schätzung:** 5.0 - -**Description:** The Prometheus metrics endpoint is publicly accessible without authentication: - -```go -r.GET("/metrics", gin.WrapH(promhttp.Handler())) -``` - -**Impact:** Exposes internal metrics including request rates, error rates, latency distributions, and potentially Go runtime metrics, providing reconnaissance data to attackers. - -**Recommendation:** -- Add authentication (admin token or basic auth) to `/metrics` -- Or restrict access via network-level controls and document this requirement - ---- - -#### M6. No Request Timeout Middleware - -**Location:** `internal/controller/setup.go` -**CVSS-Schätzung:** 5.0 - -**Description:** A TODO exists for request timeout middleware that was never implemented: - -```go -// TODO: timeout middleware -``` - -**Impact:** Long-running requests (e.g. complex queries, large file uploads, slow hooks) can tie up server goroutines indefinitely, enabling resource exhaustion attacks. - -**Recommendation:** Implement `http.TimeoutHandler` or Gin timeout middleware with configurable duration (e.g. 30s default, configurable per collection). - ---- - -#### M7. JWT Token Contains Full User Object - -**Location:** `internal/models/login.go:95-103` -**CVSS-Schätzung:** 4.3 - -**Description:** The JWT access token includes the full user object (minus password), including arbitrary `meta` data: - -```go -userCopy := *user -userCopy.InsertTime = nil -userCopy.UpdateTime = nil -userCopy.Meta = nil -tokenData = &userCopy -``` - -The `Meta` is set to `nil`, but `Permissions`, `LLMBudgets`, and other fields are included. The token is not encrypted, only signed. - -**Impact:** Token leakage exposes user metadata and internal permission structures. Tokens are also long-lived (60 minutes) without revocation support. - -**Recommendation:** -- Minimize token claims to `{id, username, role}` -- Load full user data from DB when needed -- Consider shorter token lifetime with more frequent refresh -- Implement token revocation (e.g. via token blacklist or `iat` validation) - ---- - -### LOW - -#### L1. Typo in Log Output (Unchanged) - -**Location:** `internal/controller/setup.go:62` - -**Description:** Log message reads `"senty gin activated"` instead of `"sentry gin activated"`. - ---- - -#### L2. Weak Password Policy - -**Location:** `internal/models/user.go:38` -**CVSS-Schätzung:** 3.7 - -**Description:** Password minimum length is 3 characters: - -```go -Password *string `json:"password,omitempty" bson:"password" validator:"required,min=3"` -``` - -**Impact:** Extremely weak passwords are permitted, compounding the brute-force risk. - -**Recommendation:** Increase minimum to 8+ characters. Consider complexity requirements. - ---- - -#### L3. `allowUpperPaths: true` in Default Config - -**Location:** `config.yml:15` - -**Description:** The default config allows traversal above config directory: - -```yaml -security: - allowAbsolutePaths: false - allowUpperPaths: true -``` - -This enables `fs.readFile("../../etc/passwd")` from hooks (though constrained by suexec UID). - -**Recommendation:** Default to `false` for `allowUpperPaths` and only enable when explicitly needed. - ---- - -#### L4. Hook Cookie Functions Without Scope Restrictions - -**Location:** `internal/hook/context_cookie.go` - -**Description:** The `cookie.set` function in hooks can set cookies with arbitrary names, paths, domains, and security flags. A malicious hook could overwrite authentication cookies or set cookies for different domains. - -**Recommendation:** Document this as a trust boundary — hook code has the same trust level as server-side code. Consider restricting cookie domain to the current request's domain. - ---- - -#### L5. LLM Quota Race Condition - -**Location:** `internal/llm/quota.go:61-104` - -**Description:** LLM budget checking and recording happens in two separate operations (read, then write). Under concurrent requests, a user could exceed their token budget before the write propagates: - -```go -// 1. Check budget (read) -if err := llm.CheckBudget(budgets, req.Provider); err != nil { ... } -// 2. Use tokens (API call happens here) -// 3. Record usage (write) -_ = qm.RecordUsage(auth.ID, req.Provider, resp.Model, usage, budgets) -``` - -**Impact:** Minor over-budget usage under high concurrency; not exploitable for significant financial impact. - -**Recommendation:** Accept current behavior with documentation, or use MongoDB atomic `$inc` for budget tracking. - ---- - -### RESOLVED - -#### ~~R1. Commented-Out Collection Count Endpoint~~ - -**Location:** `internal/controller/setup.go`, `internal/controller/collection_count.go` - -The dedicated `GET /_/:project/:collection/_count` endpoint is now implemented with full permission pipeline. - ---- - -## Security Posture Assessment - -### Strengths ✅ - -1. **MongoDB Operator Whitelist:** Filter operators are validated against an explicit allowlist (`filter_valid_functions.go`), blocking dangerous operators like `$where` -2. **Suexec Process Isolation:** Projects run with dedicated UID/GID on Linux, providing filesystem-level sandboxing -3. **Permission System Depth:** Multi-layered permission model (public → token → user → custom) with pre-filters and projection restrictions -4. **Hook Timeout Support:** JS hooks have configurable timeouts to prevent infinite loops -5. **Audit Logging:** Comprehensive audit trail for admin actions with token identification -6. **Bulk Operation Guard:** Requires explicit `bulk: true` config, preventing accidental mass operations -7. **File Path Sanitization:** `CleanFilePath` provides directory traversal protection with configurable strictness -8. **Error Code Tracking:** Each error gets a unique code for correlation without exposing internals (when `response.error.message` is configured) -9. **Env-Var Substitution:** Secrets can be externalized via `${ENV_VAR}` syntax -10. **Admin Token Granularity:** Fine-grained permission system with wildcards for server admin tokens - -### Weaknesses ❌ - -1. **Cryptographic:** ~~MD5 password hashing~~ ✅ behoben (bcrypt), no salt ✅ behoben (bcrypt nutzt eingebauten Salt), weak password policy -2. **Network Security:** No TLS termination (expected to be behind proxy), ~~no rate limiting~~ ✅ behoben, ~~Cookie Secure=false~~ ✅ behoben (konfigurierbar, Default: true) -3. **Token Security:** ~~Timing-attack-vulnerable comparisons~~ ✅ behoben (constant-time compare), query parameter leakage ℹ️ akzeptiertes Risiko, no revocation -4. **Input Validation Gaps:** No file upload size limits, filename not sanitized, ReDoS via `$regex` -5. **Execution Safety:** Unrestricted shell commands and HTTP fetch in hooks (SSRF) -6. **Missing Middleware:** No request timeout, ~~no rate limiting~~ ✅ behoben (Login-Rate-Limiting), no body size limits - ---- - -## Prioritized Remediation Plan - -### Phase 1 — Critical (Immediate, 1-2 Weeks) - -| # | Finding | Effort | Impact | -|---|---------|--------|--------| -| ~~C1~~ | ~~Replace MD5 with bcrypt + migration path~~ | ~~Medium~~ | ✅ **ERLEDIGT** — bcrypt + MD5-Migration implementiert | -| ~~C2~~ | ~~Add login rate limiting~~ | ~~Low~~ | ✅ **ERLEDIGT** — Exponentielles Backoff Rate Limiting (IP + Username) implementiert | -| ~~C3~~ | ~~Use `subtle.ConstantTimeCompare` for tokens~~ | ~~Low~~ | ✅ **ERLEDIGT** — Alle Token-Vergleiche auf constant-time umgestellt | -| H1 | Remove hardcoded JWT secret from config.yml | Low | Prevents token forgery from source | - -### Phase 2 — High (2-4 Weeks) - -| # | Finding | Effort | Impact | -|---|---------|--------|--------| -| H3 | Make cookie `Secure` flag configurable | Low | ✅ **ERLEDIGT** — `api.secureCookies` Config-Eintrag (Default: true) | -| L2 | Increase minimum password length to 8 | Low | Reduces brute-force surface | -| M1 | Default to generic error messages | Low | Prevents info disclosure | -| M6 | Implement request timeout middleware | Medium | Prevents resource exhaustion | - -### Phase 3 — Medium (1-2 Months) - -| # | Finding | Effort | Impact | -|---|---------|--------|--------| -| ~~H2~~ | ~~Deprecate query-parameter tokens~~ | ~~Medium~~ | ℹ️ **AKZEPTIERTES RISIKO** — bewusste Design-Entscheidung, Timing-Fix angewendet | -| H5 | Add SSRF protection for http.fetch | Medium | Blocks internal network access | -| H6 | Add exec command whitelist config | Medium | Limits command execution | -| M2 | Add file upload size/type validation | Medium | Prevents resource exhaustion | -| M3 | Sanitize uploaded filenames | Low | Prevents path manipulation | -| M4 | Add regex complexity limits | Medium | Prevents ReDoS on MongoDB | -| M5 | Protect /metrics endpoint | Low | Hides internal metrics | -| M7 | Minimize JWT token claims | Medium | Reduces exposure surface | - -### Phase 4 — Hardening (Ongoing) - -| # | Finding | Effort | Impact | -|---|---------|--------|--------| -| L3 | Default `allowUpperPaths` to false | Low | Tighter default security | -| L4 | Document hook trust boundaries | Low | Operational awareness | -| L5 | Atomic LLM budget tracking | Low | Prevents minor over-use | - ---- - -## Security Best Practices - -### Configuration - -```yaml -api: - jwtSecret: ${JWT_SECRET} # Always use env vars for secrets - cors: - allowOrigins: - - https://yourdomain.com # Never use "*" in production - allowCredentials: true - -security: - allowAbsolutePaths: false - allowUpperPaths: false # Restrict to config directory - -response: - error: - message: "An error occurred" # Generic error for production - stack: false # Never in production -``` - -### Permissions - -```yaml -permissions: - public: - methods: - get: true # Minimal public access - filter: - published: true # Always pre-filter public data - user: - methods: - get: true - post: true - put: true - filter: - author: - $subst: - expr: "auth.id" # Users only see own documents - validProjections: - - summary # Restrict visible fields -``` - -### Token Permissions - -```yaml -permissions: - "x-api-key:${API_KEY}": # Use env vars for token values - methods: - get: true - filter: - public: true -``` - -### Hooks Security - -```js -// Always validate input -var req = request(); -var body = json.parse(req.body()); - -if (!body.email || body.email.indexOf("@") === -1) { - throw { status: 400, json: { error: "invalid_email" } }; -} - -// Use bcrypt for password hashing in custom auth flows -var hash = bcrypt.hash(body.password, { cost: 12 }); - -// Sanitize exec commands -var allowedCommands = ["convert", "ffmpeg", "python3"]; -if (allowedCommands.indexOf(command) === -1) { - throw { status: 400, json: { error: "command_not_allowed" } }; -} -``` diff --git a/src/content/docs/server/15-troubleshooting.md b/src/content/docs/server/15-troubleshooting.md deleted file mode 100644 index 90fdc94..0000000 --- a/src/content/docs/server/15-troubleshooting.md +++ /dev/null @@ -1,232 +0,0 @@ ---- -title: 'Troubleshooting & FAQ' ---- - -# Troubleshooting & FAQ - -## Common Errors - -### Connection Errors - -**Error:** `failed to connect to MongoDB` - -- Check that MongoDB is running and reachable -- Verify `api.mongoUri` in `config.yml` or `API_MONGOURI` env var -- For Docker: ensure the MongoDB container is on the same network -- For replica sets: ensure all members are reachable - -**Error:** `server selection timeout` - -- MongoDB replica set may not be initialized -- Run: `rs.initiate()` in the MongoDB shell - ---- - -### Authentication Errors - -**Error:** `401 Unauthorized` on all requests - -- Check that `X-Auth-Token` header is set correctly -- Verify JWT is not expired (60 min default) -- Ensure `api.jwtSecret` matches between token generation and validation - -**Error:** `403 Forbidden` on collection access - -- Check collection permissions in project `config.yml` -- Verify user is assigned to the project (`project.users[]`) -- Check if token-based permission has correct value (wrong value = immediate deny) -- Admin role (0) bypasses all permission checks - -**Error:** Login returns wrong password even though it's correct - -- Passwords are stored as MD5 hashes -- Check if password was stored in plaintext vs. hashed -- Use `crypto.md5("password")` in a hook to verify the expected hash - ---- - -### Hook Errors - -**Error:** `ReferenceError: require is not defined` - -- Hooks run in goja (ES5.1+), not Node.js -- There is no `require()`, `import`, or npm -- Use the built-in context packages (`db`, `http`, `fs`, etc.) - -**Error:** `ReferenceError: async is not defined` - -- goja does not support `async/await` -- All hook functions are synchronous -- Use `http.fetchStream` for streaming, not async patterns - -**Error:** `TypeError: X is not a function` - -- Check the function name — it may have been renamed -- See the [legacy compatibility table](06-hooks.md#legacy-compatibility) -- Common: `readCollection` → `db.find`, `mail` → `smtp.sendMail` - -**Error:** JS hook timeout - -- Increase timeout in `hookConfig.timeout` (milliseconds) -- For streaming hooks, use `channel.subscribe()` which auto-disables timeout -- Check for infinite loops in hook code - -**Error:** Hook stopped after client disconnected - -- By default, hooks follow the incoming HTTP request lifecycle -- Set `ignoreRequestCancel: true` on the hook config only for critical hooks that must continue after client disconnect -- `ignoreRequestCancel` does not disable the hook timeout; increase `timeout` separately if needed -- `DisableTimeout()` only removes the execution timeout for specific built-in long-running operations; it does not, by itself, ignore request cancellation - -**Error:** Hook exception not returning expected format - -- Ensure the thrown object has the right keys: `status`, `json`, `html`, `string`, `bytes`, `file` -- If `response.write()` was already called, exceptions can only abort — no new response body - ---- - -### File Upload Errors - -**Error:** Files not being saved - -- Check that the upload path exists and is writable -- File fields must use Base64 data URI format: `data:mime/type;base64,...` -- Field type must be `file` or `file[]` in collection config - -**Error:** Image filter returns 404 - -- Verify the filter name exists in `imageFilter` config -- Check file path: `/{namespace}/{collection}/{id}/{field}/{filename}?filter={name}` -- Use `?no-cache=1` to force regeneration - ---- - -### Configuration Errors - -**Error:** Project not loading - -- Check YAML syntax in project `config.yml` -- Verify the project directory is under `api.projectsDir` -- Check file permissions (readable by server process) -- Use admin reload: `POST /api/v1/_/{project}/_/admin/reload` - -**Error:** Environment variable not resolved - -- Format must be `${VAR_NAME}` (with dollar and braces) -- Variable must be set in the server's environment -- Env file path must be correct in `config.yml` - -**Error:** CORS not working - -- CORS is configured at three levels: server → project → collection -- Check if `merge: true` is set for inheriting parent CORS -- `allowOrigins: ["*"]` does not work with `allowCredentials: true` -- Note: `exposeHeaders` always replaces (not appends) even with `merge: true` (known bug) - ---- - -## Debugging Tools - -### Server-Side Debugging - -```js -// Print to stdout -context.debug.dump(context.data, "some label"); - -// Get Sentry trace ID for error tracking -var traceId = debug.sentryTraceId(); - -// Pause execution -context.debug.sleep(5000); // 5 seconds -``` - -### Request Inspection - -```js -var req = request(); -context.debug.dump({ - method: req.method, - path: req.path, - url: req.url, - host: req.host, - clientIp: req.clientIp, - authHeader: req.header("Authorization"), - body: req.body() -}); -``` - -### Database Debugging - -```js -// Check what's in a collection -var all = db.find("posts", { limit: 5 }); -context.debug.dump(all); - -// Count documents -var count = db.count("posts", { - filter: { status: "published" } -}); -context.debug.dump("Published posts:", count); -``` - -### HTTP Debugging - -Use the `test.http` file in the project root for quick API testing with VS Code's REST Client extension. - ---- - -## FAQ - -### Can I use TypeScript in hooks? - -No. Hooks run in goja (ECMAScript 5.1+). However, the `tibi-types` package provides TypeScript definitions for IDE autocompletion during development. - -### Can I use async/await? - -No. The goja runtime is synchronous. Use `http.fetchStream` for streaming and `channel.subscribe` for real-time patterns. - -### How do I handle file uploads? - -Embed files as Base64 data URIs in the JSON body. Set the field type to `file` or `file[]`. The server automatically extracts and saves the file, replacing the URI with a file path. - -### How do I serve static files? - -Configure `assets` in your project config with a name and path, then access via `/api/v1/{namespace}/assets/{name}/{path}`. - -### Can I use WebSockets? - -No. Tibi-Server uses Server-Sent Events (SSE) for real-time communication. Use `channel.subscribe` in a hook for the server side and `EventSource` on the client. - -### How do I run database migrations? - -Use hooks or jobs. Create a `post.before` hook on a migration collection, or a cron job that checks and applies necessary schema changes. - -### How do I reset a user's password? - -As admin, update the user via PUT. The password will be automatically MD5-hashed on save. - -### How do I add custom endpoints? - -Use hooks with custom response writing. A `get.before` hook that calls `response.write()` and throws effectively creates a custom endpoint. - -### What happens if a hook throws without a status? - -Default status is `500`. The error message is included in a JSON error response. - -### How do I keep a critical hook running after the client disconnects? - -Set `ignoreRequestCancel: true` on that hook and configure a suitable `timeout`. - -Use this only for truly critical side effects such as order submission or writing to an outbox. For larger workflows, prefer persisting work and continuing asynchronously in a job instead of holding the HTTP request open. - -### How do I debug CORS issues? - -1. Check browser DevTools → Network → look for preflight (OPTIONS) requests -2. Verify `allowOrigins` includes your domain (exact match, not regex) -3. Check if `allowCredentials` is needed (cookies, auth headers) -4. Verify `allowMethods` includes the HTTP method you're using -5. Remember: wildcard origin `*` does not work with credentials - -### How do I scale horizontally? - -Currently, tibi-server is designed for single-instance deployment. Realtime channels are in-memory and not synchronized across instances. For multi-instance setups, put a load balancer in front and ensure sticky sessions for SSE connections. diff --git a/src/content/docs/server/16-cookbook.md b/src/content/docs/server/16-cookbook.md deleted file mode 100644 index 86096db..0000000 --- a/src/content/docs/server/16-cookbook.md +++ /dev/null @@ -1,686 +0,0 @@ ---- -title: 'Collection Config Cookbook' ---- - -# Collection Config Cookbook - -Ready-to-use collection configuration recipes for common use cases. - ---- - -## Blog with Drafts and Publishing - -```yaml -collections: - - name: posts - fields: - - key: title - type: string - validation: - required: true - - key: slug - type: string - validation: - required: true - - key: body - type: string - - key: excerpt - type: string - - key: coverImage - type: file - - key: tags - type: string[] - - key: author - type: string - - key: authorName - type: string - - key: status - type: string - validation: - required: true - in: - - draft - - published - - archived - - key: publishedAt - type: date - permissions: - public: - methods: - get: true - filter: - status: published - user: - methods: - get: true - post: true - put: true - delete: true - projections: - summary: - title: 1 - slug: 1 - excerpt: 1 - coverImage: 1 - tags: 1 - authorName: 1 - publishedAt: 1 - full: - title: 1 - slug: 1 - body: 1 - excerpt: 1 - coverImage: 1 - tags: 1 - author: 1 - authorName: 1 - status: 1 - publishedAt: 1 - hooks: - post.before: hooks/posts/before_create.js - imageFilter: - thumbnail: - - width: 400 - height: 300 - fit: true - quality: 80 - outputType: webp - hero: - - width: 1200 - height: 600 - fill: true - anchor: center - quality: 85 - outputType: webp - indexes: - - keys: - slug: 1 - options: - unique: true - - keys: - status: 1 - publishedAt: -1 - audit: - enabled: true -``` - -**Hook: `hooks/posts/before_create.js`** -```js -var auth = user.auth(); -context.data.author = auth.id; -context.data.authorName = auth.username; - -if (!context.data.status) { - context.data.status = "draft"; -} - -if (context.data.status === "published" && !context.data.publishedAt) { - context.data.publishedAt = new Date().toISOString(); -} -``` - ---- - -## User Profiles (Self-Managed) - -```yaml -collections: - - name: profiles - fields: - - key: userId - type: string - validation: - required: true - - key: displayName - type: string - - key: bio - type: string - - key: avatar - type: file - - key: website - type: string - - key: social - type: object - fields: - - key: twitter - type: string - - key: github - type: string - permissions: - public: - methods: - get: true - user: - methods: - get: true - post: true - put: true - filter: - userId: - $subst: - expr: "auth.id" - projections: - public: - displayName: 1 - bio: 1 - avatar: 1 - website: 1 - social: 1 - indexes: - - keys: - userId: 1 - options: - unique: true - imageFilter: - small: - - width: 100 - height: 100 - fill: true - anchor: center - quality: 80 - outputType: webp -``` - ---- - -## E-Commerce Products with Variants - -```yaml -collections: - - name: products - fields: - - key: name - type: string - validation: - required: true - - key: description - type: string - - key: price - type: float - validation: - required: true - - key: currency - type: string - validation: - in: [EUR, USD, GBP] - - key: images - type: file[] - - key: category - type: string - - key: tags - type: string[] - - key: variants - type: object[] - fields: - - key: name - type: string - - key: sku - type: string - - key: price - type: float - - key: stock - type: int - - key: active - type: bool - permissions: - public: - methods: - get: true - filter: - active: true - "x-api-key:${SHOP_API_KEY}": - methods: - get: true - post: true - put: true - delete: true - projections: - catalog: - name: 1 - price: 1 - currency: 1 - images: 1 - category: 1 - tags: 1 - indexes: - - keys: - category: 1 - - keys: - tags: 1 - - keys: - active: 1 - category: 1 - imageFilter: - thumb: - - width: 300 - height: 300 - fit: true - quality: 85 - outputType: webp - large: - - width: 1200 - height: 1200 - fit: true - quality: 90 -``` - ---- - -## Chat Messages with Realtime - -```yaml -collections: - - name: messages - fields: - - key: room - type: string - validation: - required: true - - key: sender - type: string - - key: senderName - type: string - - key: text - type: string - validation: - required: true - - key: type - type: string - validation: - in: [text, image, system] - permissions: - user: - methods: - get: true - post: true - hooks: - post.before: hooks/messages/before_send.js - post.after: hooks/messages/after_send.js - indexes: - - keys: - room: 1 - insertTime: -1 - - - name: chat-stream - hooks: - get.before: hooks/messages/stream.js - permissions: - user: - methods: - get: true -``` - -**Hook: `hooks/messages/before_send.js`** -```js -var auth = user.auth(); -context.data.sender = auth.id; -context.data.senderName = auth.username; -if (!context.data.type) { - context.data.type = "text"; -} -``` - -**Hook: `hooks/messages/after_send.js`** -```js -context.channel.send("chat-" + context.data.room, { - id: context.data.id, - sender: context.data.sender, - senderName: context.data.senderName, - text: context.data.text, - type: context.data.type, - time: context.data.insertTime -}); -``` - -**Hook: `hooks/messages/stream.js`** -```js -var req = request(); -var room = req.query("room"); -if (!room) { - throw { status: 400, json: { error: "room parameter required" } }; -} - -context.response.header("Content-Type", "text/event-stream"); -context.response.header("Cache-Control", "no-cache"); -context.response.writeHeader(200); - -var sub = channel.subscribe("chat-" + room, { - bufferSize: 200, - lastN: 50 -}); - -var msg; -while ((msg = sub.receive()) !== null) { - context.response.writeSSE("message", context.json.stringify(msg)); -} -sub.close(); -throw { status: 200 }; -``` - ---- - -## File Storage with Metadata - -```yaml -collections: - - name: documents - fields: - - key: title - type: string - validation: - required: true - - key: file - type: file - validation: - required: true - - key: mimeType - type: string - - key: size - type: int - - key: uploadedBy - type: string - - key: folder - type: string - - key: tags - type: string[] - permissions: - user: - methods: - get: true - post: true - put: true - delete: true - hooks: - post.before: hooks/documents/before_upload.js - indexes: - - keys: - folder: 1 - - keys: - tags: 1 - - keys: - uploadedBy: 1 -``` - -**Hook: `hooks/documents/before_upload.js`** -```js -var auth = user.auth(); -context.data.uploadedBy = auth.id; - -// Extract mime type from data URI -if (context.data.file && context.data.file.indexOf("data:") === 0) { - var mime = context.data.file.split(";")[0].split(":")[1]; - context.data.mimeType = mime; -} -``` - ---- - -## Cross-Referenced Collections - -```yaml -collections: - - name: authors - fields: - - key: name - type: string - validation: - required: true - - key: email - type: string - - key: avatar - type: file - - - name: articles - fields: - - key: title - type: string - validation: - required: true - - key: body - type: string - - key: authorId - type: string - validation: - required: true - - key: categoryIds - type: string[] - hooks: - get.after: hooks/articles/after_get.js - - - name: categories - fields: - - key: name - type: string - - key: slug - type: string -``` - -**Hook: `hooks/articles/after_get.js`** -```js -// Resolve author references -var authorIds = []; -for (var i = 0; i < context.result.length; i++) { - if (context.result[i].authorId && authorIds.indexOf(context.result[i].authorId) === -1) { - authorIds.push(context.result[i].authorId); - } -} - -if (authorIds.length > 0) { - var authors = db.find("authors", { - filter: { id: { "$in": authorIds } }, - selector: { name: 1, avatar: 1 } - }); - - var authorMap = {}; - for (var j = 0; j < authors.length; j++) { - authorMap[authors[j].id] = authors[j]; - } - - for (var k = 0; k < context.result.length; k++) { - context.result[k].author = authorMap[context.result[k].authorId] || null; - } -} -``` - ---- - -## Event Calendar - -```yaml -collections: - - name: events - fields: - - key: title - type: string - validation: - required: true - - key: description - type: string - - key: location - type: string - - key: startDate - type: date - validation: - required: true - - key: endDate - type: date - - key: allDay - type: bool - - key: recurring - type: string - validation: - in: ["", daily, weekly, monthly, yearly] - - key: image - type: file - - key: organizer - type: string - - key: maxParticipants - type: int - - key: currentParticipants - type: int - permissions: - public: - methods: - get: true - filter: - startDate: - $gte: - $subst: - eval: "new Date().toISOString()" - hooks: - put.before: hooks/events/register.js - indexes: - - keys: - startDate: 1 - - keys: - location: 1 -``` - ---- - -## Settings / Key-Value Store - -```yaml -collections: - - name: settings - fields: - - key: key - type: string - validation: - required: true - - key: value - type: string - - key: group - type: string - permissions: - "x-admin-token:${SETTINGS_TOKEN}": - methods: - get: true - post: true - put: true - delete: true - user: - methods: - get: true - indexes: - - keys: - key: 1 - options: - unique: true - - keys: - group: 1 -``` - ---- - -## Bulk Operations - -### Bulk Update — Change Category for Multiple Documents - -Update all products in category "old" to category "new": - -```bash -curl -X PUT \ - "https://example.com/api/v1/_/myproject/products?filter={\"category\":\"old\"}" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"category": "new"}' -``` - -### Bulk Update — Increase Prices by 10% - -Use the `$mul` operator to increase all prices by 10%: - -```bash -curl -X PUT \ - "https://example.com/api/v1/_/myproject/products?filter={\"category\":\"premium\"}" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"price": {"$mul": 1.1}}' -``` - -### Bulk Update — Decrement Stock - -Use `$inc` with a negative value to reduce stock for all matched items: - -```bash -curl -X PUT \ - "https://example.com/api/v1/_/myproject/products?filter={\"warehouse\":\"A\"}" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"stock": {"$inc": -5}}' -``` - -### Bulk Update — Remove a Field - -Use `$unset` to remove a field from all matched documents: - -```bash -curl -X PUT \ - "https://example.com/api/v1/_/myproject/products?filter={\"deprecated\":true}" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d '{"legacyField": {"$unset": true}}' -``` - -### Bulk Delete — Delete by Filter - -Delete all archived posts: - -```bash -curl -X DELETE \ - "https://example.com/api/v1/_/myproject/posts?filter={\"status\":\"archived\"}" \ - -H "Authorization: Bearer $TOKEN" -``` - -### Optimised Bulk Update with Hook - -For large datasets, use a `bulkUpdate` hook step with `context.db.updateMany()` for a single database call instead of per-document processing: - -```yaml -collections: - - name: products - hooks: - put.bulkUpdate: hooks/products/bulk_update.js - put.return: hooks/products/bulk_return.js -``` - -**hooks/products/bulk_update.js:** -```javascript -// Optimised: single DB call to update all matching documents -var filter = context.filter; -var body = context.body; - -var result = context.db.updateMany("products", filter, { changes: body }); -context.log("Updated " + result.modifiedCount + " documents"); -``` - -### Optimised Bulk Update with Operators in Hook - -Use the `update` parameter instead of `changes` to pass raw MongoDB update operators: - -```javascript -// Increase price by 10% for all matching products -var filter = context.filter; -var result = context.db.updateMany("products", filter, { - update: { "$mul": { "price": 1.1 } } -}); -context.log("Updated " + result.modifiedCount + " documents"); -``` - -### Optimised Bulk Delete with Hook - -```yaml -collections: - - name: products - hooks: - delete.bulkDelete: hooks/products/bulk_delete.js - delete.return: hooks/products/bulk_return.js -``` - -**hooks/products/bulk_delete.js:** -```javascript -var filter = context.filter; -var result = context.db.deleteMany("products", filter); -context.log("Deleted " + result.deletedCount + " documents"); -``` diff --git a/src/content/docs/server/17-field-level-permissions.md b/src/content/docs/server/17-field-level-permissions.md deleted file mode 100644 index 6609651..0000000 --- a/src/content/docs/server/17-field-level-permissions.md +++ /dev/null @@ -1,550 +0,0 @@ ---- -title: 'Field-Level Permissions' ---- - -# Field-Level Permissions - -## Überblick - -Field-Level Permissions erlauben den Zugriff auf **einzelne Felder** innerhalb einer Collection einzuschränken. Das System verwendet ein **3-Layer-Kaskadenmodell** mit **Fehler-Modus** (400 Bad Request) statt stillem Stripping. - -Zusätzlich können `readonly` und `hidden` auf Field-Definition-Ebene als **Eval-Ausdrücke** (JavaScript) definiert werden, die **pro Dokument** dynamisch ausgewertet werden. - -**Zwei Feld-Typen:** - -| Typ | POST/PUT | GET | Config | -|-----|----------|-----|--------| -| `readonlyFields` | **400-Fehler** wenn im Body | Normal sichtbar | Felder bleiben in `fields[]`; Liste in `yourPermissions` | -| `hiddenFields` | **400-Fehler** wenn im Body | Aus Response **gestripped** | Felder aus `fields[]` **entfernt**; Liste nicht in `yourPermissions` | - -**Admin-Bypass:** Admin (role=0) und Admin-Token → `GetEffectivePermissionSet()` gibt `nil` zurück → keine Einschränkungen. - ---- - -## Konfiguration - -### Drei Ebenen (Kaskade) - -```yaml -name: orders - -# Ebene 1: Collection-Level (Basis für alle PermissionSets) -readonlyFields: [createdBy] -hiddenFields: [costPrice] - -permissions: - user: - methods: - get: true - post: true - put: true - # Ebene 2: PermissionSet-Level (addiert/subtrahiert von Collection-Level) - readonlyFields: [customerName] - hiddenFields: [internalMargin] - -fields: - - name: alwaysReadonly - type: number - readonly: true # Ebene 3: Field-Definition (absolutes Override) - - - name: alwaysHidden - type: string - hidden: true # Ebene 3: Field-Definition (absolutes Override) - - - name: neverReadonly - type: string - readonly: false # Ebene 3: entfernt Feld aus JEDEM effektiven readonly-Set - - - name: conditionalReadonly - type: string - readonly: - eval: "$.status === 'confirmed'" # Ebene 3: Eval — readonly nur wenn Bedingung zutrifft - - - name: conditionalHidden - type: string - hidden: - eval: "$.status === 'confirmed'" # Ebene 3: Eval — hidden nur wenn Bedingung zutrifft -``` - -### Kaskadenmodell im Detail - -Die effektive Feldliste wird in 4 Schritten berechnet (`ResolveEffectiveFields`): - -1. **Collection-Level** → Basis-Set: `readonlyFields: [createdBy]` -2. **PermissionSet-Level** → Hinzufügen/Entfernen mit `-` Prefix-Negation: - - `readonlyFields: [customerName]` → fügt `customerName` hinzu - - `readonlyFields: [-createdBy]` → entfernt `createdBy` aus dem Set -3. **Parent-Expansion** → Elternfelder werden auf alle Sub-Felder expandiert (z.B. `payment` → `payment.method`, `payment.cardNumber`) -4. **Field-Definition Override** (letztes Wort): - - `readonly: true` → Feld wird **immer** hinzugefügt (egal ob in Liste oder nicht) - - `readonly: false` → Feld wird **immer** entfernt (überschreibt alles) - - `hidden: true` / `hidden: false` → analog für Hidden-Felder - - `readonly: {eval: "..."}` / `hidden: {eval: "..."}` → **Eval-Ausdruck**, wird pro Dokument ausgewertet (siehe [Eval-Ausdrücke](#eval-ausdrücke-dynamische-feld-permissions)) - -### Ergebnis-Beispiel - -Mit der obigen Config ergibt sich: - -| Feld | user effektiv readonly | user effektiv hidden | -|------|----------------------|---------------------| -| createdBy | ✅ (Collection-Level) | — | -| customerName | ✅ (PermissionSet) | — | -| alwaysReadonly | ✅ (Field-Def `readonly: true`) | — | -| neverReadonly | ❌ (Field-Def `readonly: false` überschreibt) | — | -| costPrice | — | ✅ (Collection-Level) | -| internalMargin | — | ✅ (PermissionSet) | -| alwaysHidden | — | ✅ (Field-Def `hidden: true`) | - -### Verschachtelung (Dot-Notation) - -Felder in verschachtelten Objekten werden per Dot-Notation adressiert: - -```yaml -hiddenFields: - - payment.cardNumber - - payment.cvv -readonlyFields: - - address.zip -``` - -Wenn ein **Elternfeld** gelistet wird, werden automatisch **alle Sub-Felder** mit einbezogen (es sei denn, ein Sub-Feld hat ein Field-Def-Override `false`). - -### Token-basierte Permissions - -Tokens erhalten ein eigenes PermissionSet und unterliegen denselben Feld-Einschränkungen: - -```yaml -permissions: - "token:MYTOKEN": - methods: - get: true - post: true - readonlyFields: [status] - # Erbt costPrice von Collection-Level hiddenFields -``` - ---- - -## Eval-Ausdrücke (Dynamische Feld-Permissions) - -### Überblick - -Neben statischem `readonly: true/false` und `hidden: true/false` kann auf Field-Definition-Ebene ein **JavaScript-Eval-Ausdruck** definiert werden. Dieser wird **pro Dokument** ausgewertet und ermöglicht konditionale Feld-Permissions basierend auf Dokumentinhalt, User-Kontext oder HTTP-Methode. - -**Wichtig:** Eval-Ausdrücke nehmen **nicht** an der statischen Kaskade (Layer 1–3) teil. Sie werden als **separater Post-Pass** nach den statischen Checks ausgeführt. - -### Syntax - -```yaml -fields: - - name: confirmedAt - type: string - readonly: - eval: "$.status === 'confirmed'" # JS-Ausdruck → readonly wenn true - - - name: secretNote - type: string - hidden: - eval: "$.status === 'confirmed'" # JS-Ausdruck → hidden wenn true -``` - -### Verfügbare Kontext-Variablen - -| Variable | Typ | Beschreibung | -|----------|-----|-------------| -| `$` | `object` | Das vollständige Dokument (bei POST: der Body; bei PUT: das bestehende DB-Dokument + Body-Merge; bei GET: das DB-Dokument) | -| `$this` | `any` | Der Wert des aktuellen Felds im Dokument | -| `$auth` | `object` | Der authentifizierte User (mit `id`, `username`, `role`, `permissions`) | -| `$method` | `string` | Die HTTP-Methode (`POST`, `PUT`, `GET`) | -| `$project` | `string` | Der Projektname | -| `$namespace` | `string` | Der API-Namespace | - -### Beispiele - -```yaml -# Feld readonly nach Bestätigung -readonly: - eval: "$.status === 'confirmed'" - -# Feld nur für bestimmte User hidden -hidden: - eval: "$auth.role > 1" - -# Feld readonly bei PUT, aber nicht bei POST -readonly: - eval: "$method === 'PUT'" - -# Kombination: Feld readonly wenn Betrag > 1000 -readonly: - eval: "$.amount > 1000" -``` - -### Zwei-Phasen-Architektur - -1. **Phase 1 (Statisch):** Die 3-Layer-Kaskade wird normal aufgelöst. Eval-Felder werden bei der Kaskade **ignoriert** (`.BoolValue()` gibt `nil` zurück). -2. **Phase 2 (Eval):** Nach den statischen Checks wird pro Dokument der JS-Ausdruck evaluiert. Ergibt er `true`, wird das Feld wie ein statisches readonly/hidden-Feld behandelt (400 bei POST/PUT, Strip bei GET). - -### Vorkompilierung - -Eval-Ausdrücke werden beim Laden der Config (in `AddCollection`) zu `goja.Program`-Objekten vorkompiliert. Syntaxfehler werden beim Config-Reload als Error geloggt. Zur Laufzeit wird nur noch `prog.Run()` aufgerufen. - -### Admin-Bypass - -Admin-User (role=0) und Admin-Tokens umgehen **alle** Eval-Checks. `GetEffectivePermissionSet()` gibt `nil` zurück → der gesamte Eval-Block wird übersprungen. - -### Verhalten pro Endpoint - -| Endpoint | Eval-Verhalten | -|----------|---------------| -| **POST** | Body wird als Dokument-Kontext (`$`) verwendet. Eval-readonly/hidden → 400 | -| **PUT** | Bestehendes DB-Dokument als Kontext. Eval-readonly/hidden → 400 (inkl. Operator-Bodies) | -| **GET** | Pro Ergebnis-Dokument: Eval-hidden → Feld wird aus Response gestripped | -| **Bulk-POST** | Pro Zeile individuelle Eval-Auswertung | -| **Bulk-Update** | Eval-Auswertung vor Batch-Operation | - -### Config-Auslieferung - -Eval-Definitionen (`readonly: {eval: "..."}`, `hidden: {eval: "..."}`) werden als Teil der Field-Definitionen in `fields[]` mit ausgeliefert. Der Client kann die Eval-Ausdrücke direkt aus der Config lesen und clientseitig vorvalidieren (da nur JS unterstützt wird, kann der gleiche Ausdruck im Browser ausgeführt werden). - -Für **Non-Admins** werden statisch hidden Feld-Definitionen wie gewohnt aus `fields[]` entfernt. Eval-hidden-Felder bleiben jedoch sichtbar, da sie nur per Dokumentkontext ausgewertet werden können. - ---- - -## Verhalten pro Endpoint - -### POST / Bulk-POST - -``` -POST /api/v1/_/:project/:collection -``` - -Nach Body-Parsing, **vor** Validierung und Hooks: - -1. `GetEffectivePermissionSet()` bestimmen (admin → `nil` → kein Check) -2. Effektive `readonlyFields` berechnen (3-Layer-Kaskade) -3. `CheckFieldsPresent(body, readonlyFields)` → bei Treffer: **400** `{"error": "readonly fields must not be sent", "fields": [...]}` -4. Effektive `hiddenFields` berechnen -5. `CheckFieldsPresent(body, hiddenFields)` → bei Treffer: **400** `{"error": "hidden fields must not be sent", "fields": [...]}` -6. **Eval-Phase:** `EvalFieldReadonly(body, ...)` + `EvalFieldHidden(body, ...)` → bei Treffer: **400** analog - -Bei Bulk-POST: Check wird **pro Zeile** ausgeführt. Fehlgeschlagene Zeilen werden mit Index im Error-Array zurückgegeben, erfolgreiche Zeilen werden trotzdem verarbeitet. - -### PUT / Bulk-Update - -``` -PUT /api/v1/_/:project/:collection/:id -``` - -Wie POST, zusätzlich werden **Operator-Felder** geprüft (`$set`, `$inc`, `$mul`, `$unset`, etc.): - -```json -{"$set": {"customerName": "New"}, "$inc": {"price": 10}} -``` - -→ Prüfung ob `customerName` oder `price` in readonly/hidden ist. Felder in `$set`, `$inc`, `$mul`, `$unset`, `$min`, `$max`, `$rename`, `$push`, `$pull`, `$addToSet` werden alle geprüft. - -Zusätzlich: **Eval-Phase** wie bei POST, das bestehende DB-Dokument dient als Eval-Kontext (`$`). - -### GET - -``` -GET /api/v1/_/:project/:collection -GET /api/v1/_/:project/:collection/:id -``` - -Nach DB-Read, nach Lookups, **vor** Return-Hook: - -- Hidden-Felder werden aus jedem Result-Dokument **gestripped** (`StripFields`) -- **Eval-Hidden:** Pro Dokument wird `EvalFieldHidden()` ausgewertet; bei `true` wird das Feld zusätzlich gestripped -- Readonly-Felder bleiben **sichtbar** (nur Schreibschutz, kein Leseschutz) -- Gilt für Einzel- und Listen-Abfragen - -### Config-Auslieferung (GET /api/v1/project/:id) - -Die `enrichData()`-Funktion in `project.go` liefert rollenabhängig unterschiedliche Configs: - -**Nicht-Admin (User, Public, Token):** - -- `yourPermissions[collection].readonlyFields` = effektive readonly-Liste -- `yourPermissions[collection].hiddenFields` wird **nicht** ausgeliefert -- Hidden-Feld-Definitionen werden aus `fields[]` **entfernt** (Frontend sieht sie nicht) -- Eval-Definitionen bleiben in `fields[]` sichtbar (auch eval-hidden, da nur per Dokument auswertbar) -- `hiddenFields`-Arrays werden aus Collection-Config und PermissionSets **entfernt** - -**Admin:** - -- `yourPermissions[collection].readonlyFields` wird **nicht** ausgeliefert (keine Einschränkungen) -- `yourPermissions[collection].hiddenFields` = nur als **Hint** (z.B. für UI-Markierung) -- Alle Feld-Definitionen bleiben vollständig in `fields[]` - -**Lazy-Copy-Optimierung:** Die shared API-Struct wird nur kopiert wenn tatsächlich Hidden-Felder zu strippen sind (`apiCopied`-Flag). Admin-Requests erzeugen keine Kopie. So wird Mutation des geteilten API-Objekts verhindert. - ---- - -## Vollständiges Config-Beispiel - -```yaml -name: orders -defaultLanguage: german - -# Collection-Level (Ebene 1): Basis für alle PermissionSets -readonlyFields: [createdBy] -hiddenFields: [costPrice] - -permissions: - public: - methods: - get: true - post: false - put: false - delete: false - readonlyFields: [status, customerName] - hiddenFields: [internalMargin, costPrice, payment.cardNumber] - - user: - methods: - get: true - post: true - put: true - delete: false - readonlyFields: [customerName, neverReadonly] - hiddenFields: [internalMargin] - - fulfillment: - methods: - get: true - put: true - readonlyFields: [customerName, price] - # kein hiddenFields → erbt nur Collection-Level costPrice - - "token:APITOKEN": - methods: - get: true - post: true - readonlyFields: [status] - -fields: - - name: title - type: string - validator: - required: true - - - name: status - type: string - - - name: customerName - type: string - - - name: price - type: number - - - name: internalMargin - type: number - - - name: costPrice - type: number - - - name: createdBy - type: string - - - name: alwaysReadonly - type: number - readonly: true # Field-Def: immer readonly für alle nicht-Admins - - - name: alwaysHidden - type: string - hidden: true # Field-Def: immer hidden für alle nicht-Admins - - - name: neverReadonly - type: string - readonly: false # Field-Def: NIEMALS readonly (überschreibt PermSet-Einträge) - - - name: payment - type: object - subFields: - - name: method - type: string - - name: cardNumber - type: string -``` - -### Ergebnis-Matrix (nach Kaskade) - -| Feld | admin | user | public | token:APITOKEN | fulfillment | -|------|-------|------|--------|----------------|-------------| -| title | read+write | read+write | read | read+write | read+write | -| status | read+write | read+write | **readonly** | **readonly** | read+write | -| customerName | read+write | **readonly** | **readonly** | read+write | **readonly** | -| price | read+write | read+write | read | read+write | **readonly** | -| createdBy | read+write | **readonly** | **readonly** | **readonly** | **readonly** | -| alwaysReadonly | read+write | **readonly** | **readonly** | **readonly** | **readonly** | -| neverReadonly | read+write | read+write | read+write | read+write | read+write | -| internalMargin | read+write | **hidden** | **hidden** | read+write | read+write | -| costPrice | read+write | **hidden** | **hidden** | **hidden** | **hidden** | -| alwaysHidden | read+write | **hidden** | **hidden** | **hidden** | **hidden** | -| payment.cardNumber | read+write | read+write | **hidden** | read+write | read+write | - ---- - -## Implementierung - -### Dateien - -| Datei | Zweck | -|-------|-------| -| `internal/utils/field_filter.go` | Generische Kaskaden-Engine, Check/Strip-Funktionen | -| `internal/models/field_permissions.go` | Model-spezifische Wrapper, Eval-Funktionen (Eval/Resolve/Collect) | -| `internal/models/collection.go` | Struct-Definitionen (BoolOrEval, ReadonlyFields, HiddenFields) | -| `internal/models/api.go` | Config-Laden, `compileFieldEvals()` (JS-Vorkompilierung) | -| `internal/models/permission.go` | `GetEffectivePermissionSet()` — Admin-Bypass-Logik | -| `internal/models/project.go` | `enrichData()` — Config-Auslieferung mit Lazy-Copy + Eval-Felder | -| `internal/controller/collection_post.go` | POST-Enforcement (statisch + eval) | -| `internal/controller/collection_put.go` | PUT-Enforcement (statisch + eval, inkl. Operator-Check) | -| `internal/controller/collection_get.go` | GET Hidden-Stripping (statisch + eval pro Dokument) | -| `internal/controller/collection_bulk_post.go` | Bulk-POST per-row Enforcement (statisch + eval) | -| `internal/controller/collection_bulk_update.go` | Bulk-Update Enforcement (statisch + eval) | - -### Structs - -```go -// PermissionSet (collection.go) -type PermissionSet struct { - Methods map[string]MethodPermission - Filter dbcrud.M - ValidProjections []string - ReadonlyFields []string // Felder die für dieses PermSet readonly sind - HiddenFields []string // Felder die für dieses PermSet hidden sind - CompiledFilterSubs []*utils.CompiledSubst -} - -// CollectionConfig (collection.go) -type CollectionConfig struct { - Name string - ReadonlyFields []string // Collection-Level readonly (Basis) - HiddenFields []string // Collection-Level hidden (Basis) - Fields []CollectionField - Permissions map[string]PermissionSet - // ... -} - -// CollectionField (collection.go) -type CollectionField struct { - Name string - Type string - Readonly *BoolOrEval // Field-Def: true/false (statisch) oder {eval: "..."} (dynamisch) - Hidden *BoolOrEval // Field-Def: true/false (statisch) oder {eval: "..."} (dynamisch) - SubFields []CollectionField - // ... -} - -// BoolOrEval (collection.go) — unterstützt bool oder eval-Ausdruck -type BoolOrEval struct { - Value *bool // statischer Wert (true/false) - Eval *string // JS-Ausdruck als String - JsProgram *JsProgram // vorkompiliertes goja-Programm (intern) -} - -// FieldDef (utils/field_filter.go) — import-sichere Kopie für die Engine -type FieldDef struct { - Name string - SubFields []FieldDef - Readonly *bool - Hidden *bool -} -``` - -### Kernfunktionen - -```go -// Kaskaden-Auflösung (utils/field_filter.go) -func ResolveEffectiveFields( - collectionLevel []string, - permSetLevel []string, - fields []FieldDef, - fieldDefFn func(f FieldDef) *bool, -) []string - -// Prüfung ob geschützte Felder im Body vorhanden sind -func CheckFieldsPresent(doc map[string]interface{}, fields []string) []string - -// Prüfung für Operator-Bodies ($set, $inc, etc.) -func CheckOperatorFieldsPresent(doc map[string]interface{}, fields []string) []string - -// Felder aus Response-Dokument entfernen (für GET) -func StripFields(doc map[string]interface{}, fields []string) - -// Feld-Definitionen aus Config entfernen (für enrichData) -func StripFieldDefsFromConfig(fields []FieldDef, hidden []string) []FieldDef - -// Model-Wrapper (models/field_permissions.go) -func ResolveEffectiveReadonlyFields(collConfig, permSet, fields) []string -func ResolveEffectiveHiddenFields(collConfig, permSet, fields) []string -func CollectionFieldsToFieldDefs(fields []CollectionField) []FieldDef -func StripFieldsFromConfig(fields []CollectionField, hidden []string) []CollectionField - -// Eval-Funktionen (models/field_permissions.go) -func EvalFieldReadonly(doc, collConfig, dbAPI, auth, method) ([]string, error) -func EvalFieldHidden(doc, collConfig, dbAPI, auth, method) ([]string, error) -func HasEvalFields(fields []CollectionField) bool -func HasEvalReadonlyFields(fields []CollectionField) bool -func HasEvalHiddenFields(fields []CollectionField) bool - -// Vorkompilierung (models/api.go) -func compileFieldEvals(colName, fields, moduleCache, errs) -``` - ---- - -## Abwärtskompatibilität - -- **Vollständig abwärtskompatibel**: Configs ohne `readonlyFields`/`hiddenFields` verhalten sich identisch -- `GetEffectivePermissionSet()` gibt für nicht vorkonfigurierte Felder leere Listen zurück → keine Checks -- `validProjections` arbeitet orthogonal und bleibt unberührt -- Admin-Bypass ist konsistent mit dem bestehenden Permission-System - ---- - -## Tests - -### Unit-Tests - -| Datei | Tests | Abdeckung | -|-------|-------|-----------| -| `internal/utils/field_filter_test.go` | 22 | `ResolveEffectiveFields`, `CheckFieldsPresent`, `StripFields`, `StripFieldDefsFromConfig`, `CheckOperatorFieldsPresent` | -| `internal/models/field_permissions_test.go` | 22 | `CollectionFieldsToFieldDefs`, `ResolveEffectiveReadonlyFields`, `ResolveEffectiveHiddenFields`, `StripFieldsFromConfig` | - -### Integration-Tests - -| Datei | Tests | Abdeckung | -|-------|-------|-----------| -| `internal/controller/collection_field_permission_test.go` | 31 | Alle Endpoints, alle Rollen (statische Permissions) | -| `internal/controller/collection_field_eval_test.go` | 16 | Eval-basierte Permissions (readonly + hidden) | - -**Test-Kategorien (statisch):** - -| Kategorie | Tests | Was wird geprüft | -|-----------|-------|-----------------| -| POST readonly | 6 | User/Collection-Level/Field-Def/NeverReadonly/Admin-Bypass/Erlaubte Felder | -| POST hidden | 4 | User/Collection-Level/Field-Def/Admin-Bypass | -| PUT readonly | 3 | User-Rejection/Erlaubte Felder/Admin-Bypass | -| PUT Operator | 3 | $set readonly/$inc hidden/Erlaubte $inc | -| PUT hidden | 1 | User hidden field rejection | -| GET hidden | 5 | User-Strip/Admin-Vollzugriff/Public-Strip/SingleDoc/Readonly-sichtbar | -| Token | 4 | Readonly-Rejection/Erlaubte Felder/Hidden-Rejection/GET-Strip | -| yourPermissions | 2 | User (readonlyFields ja, hiddenFields nein) / Admin (hiddenFields-Hint) | -| Config-Delivery | 2 | User (hidden Feld-Defs entfernt) / Admin (alle sichtbar) | -| Mutation-Safety | 1 | Shared API nach User-Request nicht mutiert | - -**Test-Config:** `testdata/api/testproject_fieldperm/collections/orders.yml` - -**Test-Kategorien (Eval):** - -| Kategorie | Tests | Was wird geprüft | -|-----------|-------|-----------------| -| POST eval readonly | 4 | Bedingung false=erlaubt, true=rejected, admin-bypass, ohne Feld=erlaubt | -| POST eval hidden | 2 | Bedingung false=erlaubt, true=rejected | -| POST statisch | 2 | Statisches readonly/hidden weiterhin korrekt | -| PUT eval readonly | 3 | Bedingung false=erlaubt, true=rejected, admin-bypass | -| GET eval hidden | 5 | Bedingung false=sichtbar, true=stripped, admin-sieht-alles, mixed-results, statisch-immer-stripped | - -**Test-Config:** `testdata/api/testproject_fieldperm/collections/evalorders.yml` diff --git a/src/content/docs/server/18-orgs-teams.md b/src/content/docs/server/18-orgs-teams.md deleted file mode 100644 index 7529323..0000000 --- a/src/content/docs/server/18-orgs-teams.md +++ /dev/null @@ -1,720 +0,0 @@ ---- -title: 'Orgs & Teams' ---- - -# Orgs & Teams - -## Overview - -Tibi-Server uses a 3-level hierarchy for access control and data isolation: **Org → Team → User**. Orgs define the visibility boundary (which projects a user can see), while teams define working rights (what a user can do within a project's collections). - -```text -┌─────────────────────────────────────────────────┐ -│ Org │ -│ (Visibility Boundary / Data Isolation) │ -│ │ -│ ┌──────────────┐ ┌──────────────┐ │ -│ │ Team Alpha │ │ Team Beta │ │ -│ │ (Working │ │ (Working │ │ -│ │ Rights) │ │ Rights) │ │ -│ │ │ │ │ │ -│ │ 👤 User A │ │ 👤 User B │ │ -│ │ 👤 User C │ │ 👤 User C │ ← multi │ -│ └──────┬───────┘ └──────┬───────┘ team │ -│ │ │ │ -│ ▼ ▼ │ -│ ┌──────────────────────────────┐ │ -│ │ Project X │ │ -│ │ teams: [Alpha, Beta] │ │ -│ │ orgId: this-org │ │ -│ └──────────────────────────────┘ │ -└─────────────────────────────────────────────────┘ -``` - -### Key Concepts - -| Concept | Description | -|---------|-------------| -| **Org** | Isolation boundary. Org members can see all projects of their org. | -| **Team** | Working unit. Teams are assigned to projects and carry permission sets. | -| **OrgMembership** | Links a user to an org with a role (`admin` or `member`). | -| **TeamMembership** | Links a user to a team with a role (`admin` or `member`). | -| **Project.orgId** | The owning org of a project. | -| **Project.teams[]** | Which teams have working rights on this project. | -| **User.primaryOrgId** | The user's primary org (used for LLM budget billing). | - -### Role Hierarchy - -| Role | Scope | Description | -|------|-------|-------------| -| **SystemAdmin** (`role: 0`) | Global | Full access to everything. Bypasses all access control. | -| **OrgAdmin** (`orgs[].role: "admin"`) | Per Org | Can manage teams and users within their org(s). Can assign teams to projects. | -| **OrgMember** (`orgs[].role: "member"`) | Per Org | Can see projects of their org. Working rights come from team membership. | -| **TeamAdmin** (`teams[].role: "admin"`) | Per Team | Currently same as TeamMember. Reserved for future use. | -| **TeamMember** (`teams[].role: "member"`) | Per Team | Gets collection-level working rights when team is assigned to a project. | - -## Data Model - -### Org - -```json -{ - "id": "507f1f77bcf86cd799439011", - "name": "Acme Corp", - "llmBudgets": [ - { - "provider": "openai", - "tokenBudget": 500000, - "tokensUsed": 123456, - "periodStart": "2026-01-01T00:00:00Z", - "periodEnd": "2026-04-01T00:00:00Z" - } - ], - "meta": { "industry": "Tech" }, - "insertTime": "2026-01-15T10:00:00Z", - "updateTime": "2026-01-15T10:00:00Z" -} -``` - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `name` | string | yes | Unique org name. Supports text search. | -| `llmBudgets` | LLMBudget[] | no | Org-level LLM token budgets (see [LLM Budget Cascade](#llm-budget-cascade)). | -| `meta` | object | no | Arbitrary metadata. | - -**Collection name:** `org` - -### Team - -```json -{ - "id": "507f1f77bcf86cd799439012", - "name": "Editors", - "orgId": "507f1f77bcf86cd799439011", - "permissions": ["editor", "reviewer"], - "meta": { "description": "Content editors" }, - "insertTime": "2026-01-15T10:00:00Z", - "updateTime": "2026-01-15T10:00:00Z" -} -``` - -| Field | Type | Required | Description | -|-------|------|----------|-------------| -| `name` | string | yes | Team name. Must be unique within the org. | -| `orgId` | string | yes | Reference to the parent org. Immutable after creation. | -| `permissions` | string[] | no | Named permission sets that this team grants (must match keys defined in collection configs). | -| `meta` | object | no | Arbitrary metadata. | - -**Collection name:** `team` - -**Composite identity:** `{name, orgId}` — team names are unique per org, not globally. - -### User (extended fields) - -The following fields have been added to the User model: - -```json -{ - "username": "alice", - "role": 1, - "orgs": [ - { "orgId": "507f1f77bcf86cd799439011", "role": "admin" }, - { "orgId": "507f1f77bcf86cd799439013", "role": "member" } - ], - "teams": [ - { "teamId": "507f1f77bcf86cd799439012", "role": "member" }, - { "teamId": "507f1f77bcf86cd799439014", "role": "admin" } - ], - "primaryOrgId": "507f1f77bcf86cd799439011" -} -``` - -| Field | Type | Description | -|-------|------|-------------| -| `orgs` | OrgMembership[] | Orgs the user belongs to, each with a role. | -| `teams` | TeamMembership[] | Teams the user belongs to, each with a role. | -| `primaryOrgId` | string | Primary org for LLM budget billing. Must reference an org in `orgs[]`. | - -#### OrgMembership - -| Field | Type | Values | Description | -|-------|------|--------|-------------| -| `orgId` | string | ObjectID | Reference to the org. | -| `role` | string | `"admin"`, `"member"` | User's role within this org. | - -#### TeamMembership - -| Field | Type | Values | Description | -|-------|------|--------|-------------| -| `teamId` | string | ObjectID | Reference to the team. | -| `role` | string | `"admin"`, `"member"` | User's role within this team. | - -### Project (extended fields) - -```json -{ - "name": "website", - "namespace": "website_prod", - "orgId": "507f1f77bcf86cd799439011", - "teams": ["507f1f77bcf86cd799439012", "507f1f77bcf86cd799439014"] -} -``` - -| Field | Type | Description | -|-------|------|-------------| -| `orgId` | string | The owning org. Determines which users can see this project. | -| `teams` | string[] | Team IDs assigned to this project. Members of these teams get working rights. | - -### JWT Token Claims (extended) - -The JWT access token now includes org and team membership: - -```json -{ - "object": { - "id": "60a1b2c3d4...", - "username": "alice", - "role": 1, - "permissions": [], - "orgs": [ - { "orgId": "507f1f77bcf86cd799439011", "role": "admin" } - ], - "teams": [ - { "teamId": "507f1f77bcf86cd799439012", "role": "member" } - ], - "primaryOrgId": "507f1f77bcf86cd799439011" - }, - "type": "User", - "exp": 1700000000 -} -``` - -## API Endpoints - -### Org CRUD - -All org endpoints require authentication via `X-Auth-Token`. - -#### List Orgs - -```text -GET /api/v1/org -``` - -**Visibility:** -- SystemAdmin: sees all orgs -- Others: sees only orgs they are a member of - -**Query parameters:** Standard collection query params (`_search`, `filter`, `sort`, `offset`, `limit`). - -#### Create Org - -```text -POST /api/v1/org -``` - -**Auth:** SystemAdmin only. - -**Request:** - -```json -{ - "name": "Acme Corp", - "meta": { "industry": "Tech" } -} -``` - -#### Update Org - -```text -PUT /api/v1/org/:id -``` - -**Auth:** SystemAdmin only. - -#### Delete Org - -```text -DELETE /api/v1/org/:id -``` - -**Auth:** SystemAdmin only. - -**Cascade effects on delete:** -1. All teams within the org are deleted -2. Team references are removed from all projects (`teams[]`) and users (`teams[]`) -3. Projects are decoupled: `orgId` unset, `teams` cleared -4. Org membership is removed from all users (`orgs[]`) - ---- - -### Team CRUD - -#### List Teams - -```text -GET /api/v1/team -``` - -**Visibility:** -- SystemAdmin: sees all teams -- OrgAdmin: sees all teams in orgs they admin, plus their own teams -- Others: sees only teams they are a member of - -#### Create Team - -```text -POST /api/v1/team -``` - -**Auth:** SystemAdmin or OrgAdmin of the team's org. - -**Request:** - -```json -{ - "name": "Editors", - "orgId": "507f1f77bcf86cd799439011", - "permissions": ["editor"] -} -``` - -**Validation:** -- `orgId` is required and must reference an existing org -- `orgId` is **immutable** after creation -- Team name must be unique within the org - -#### Update Team - -```text -PUT /api/v1/team/:id -``` - -**Auth:** SystemAdmin or OrgAdmin of the team's org. - -**Note:** `orgId` cannot be changed after creation. - -#### Delete Team - -```text -DELETE /api/v1/team/:id -``` - -**Auth:** SystemAdmin or OrgAdmin of the team's org. - -**Cascade effects on delete:** -1. Team reference is removed from all projects (`teams[]`) -2. Team membership is removed from all users (`teams[]`) - ---- - -### User Management (org/team-scoped) - -#### Assign User to Org - -```text -PUT /api/v1/user/:id -``` - -Add or modify org membership: - -```json -{ - "orgs": [ - { "orgId": "507f1f77bcf86cd799439011", "role": "admin" }, - { "orgId": "507f1f77bcf86cd799439013", "role": "member" } - ] -} -``` - -**Auth by role:** - -| Actor | Can Assign | Restrictions | -|-------|-----------|-------------| -| SystemAdmin | Any org, any role | None | -| OrgAdmin | Orgs they admin | Cannot modify entries for orgs outside their scope. Existing out-of-scope entries must remain unchanged. | -| Others | — | Cannot modify `orgs`. | - -#### Assign User to Team - -```text -PUT /api/v1/user/:id -``` - -```json -{ - "teams": [ - { "teamId": "507f1f77bcf86cd799439012", "role": "member" } - ] -} -``` - -**Auth:** Same rules as org assignment. OrgAdmin can only assign teams that belong to orgs they admin. - -#### Set Primary Org - -```text -PUT /api/v1/user/:id -``` - -```json -{ - "primaryOrgId": "507f1f77bcf86cd799439011" -} -``` - -**Validation:** The referenced org must be in the user's `orgs[]` array. Can be set to `null` to clear. - -#### User Visibility - -- SystemAdmin: sees all users -- Org members: see all users that share at least one org (`orgs.orgId: {$in: [myOrgIDs]}`) -- No orgs: can only see own profile - ---- - -### Project Team Assignment - -```text -PUT /api/v1/project/:id -``` - -```json -{ - "teams": ["507f1f77bcf86cd799439012", "507f1f77bcf86cd799439014"] -} -``` - -**Auth:** -- SystemAdmin: can set `orgId`, `teams`, and all other fields -- OrgAdmin: can only modify `teams[]`, and only with teams that belong to the project's org - -**Validation:** All team IDs in `teams[]` must reference teams that belong to the same org as the project (`project.orgId`). - -## Permission System - -### How Team Permissions Map to Collections - -Teams carry a `permissions[]` array of named permission set keys. These keys correspond to permission sets defined in collection configs: - -**Collection config (YAML):** - -```yaml -permissions: - public: - methods: - GET: { allow: true } - user: - methods: - GET: { allow: true } - POST: { allow: true } - editor: - methods: - GET: { allow: true } - POST: { allow: true } - PUT: { allow: true } - DELETE: { allow: true } -``` - -**Team definition:** - -```json -{ - "name": "Content Editors", - "orgId": "...", - "permissions": ["editor"] -} -``` - -When a user who is a member of the "Content Editors" team accesses a collection, they get the `editor` permission set — allowing GET, POST, PUT, and DELETE. - -### Permission Resolution Priority - -When determining access for a collection request, the following priority chain is evaluated top-to-bottom. The first match wins: - -```text -1. Admin Token (namespace-level) - └─ Checked via X-Admin-Token header with namespace.*.collection.* permissions - -2. Token Permission (query/header) - └─ Checked via ?token= or X-Collection-Token header - -3. Public Permission Set - └─ The "public" key in the collection's permissions config - -4. SystemAdmin (role=0) - └─ Always allowed — bypasses all checks - -5. Team "user" Permission - └─ If user is a member of a team assigned to this project - AND the collection has a "user" permission set → apply it - -6. Custom User Permissions - └─ User's auth.permissions[] matched against "namespace:permKey" - -7. Custom Team Permissions (NEW) - └─ Team's permissions[] matched against collection permission keys - (e.g. team has ["editor"] and collection has an "editor" set) - -8. Denied -``` - -### Effective Permission Set Selection - -For field-level access control and projections, the system selects ONE effective permission set: - -| Priority | Condition | Selected Set | -|----------|-----------|-------------| -| 1 | Admin token or SystemAdmin | `nil` (no restrictions) | -| 2 | Token permission matches | Token's permission set | -| 3 | User custom permission matches `namespace:key` | That custom permission set | -| 4 | Team permission key matches collection key | That team permission set | -| 5 | User is team member | `"user"` permission set | -| 6 | Fallback | `"public"` permission set | - -### Permission Union from Multiple Teams - -When a user is a member of multiple teams assigned to the same project, the permissions are **merged (union)**: - -```text -Team Alpha: permissions=["editor"] -Team Beta: permissions=["reviewer"] - -User in both → permissions checked: ["editor", "reviewer"] -``` - -If either team's permission set allows an operation, access is granted. - -## Visibility Rules - -### Project Visibility - -```text -SystemAdmin (role=0) → sees ALL projects -User (role=1) → sees projects where project.orgId ∈ user's org IDs -No org membership → sees NO projects -``` - -Project visibility is **org-scoped**, not team-scoped. Being a member of an org grants visibility to all projects of that org, regardless of team assignment. Team assignment grants working rights, not visibility. - -### Org Visibility - -```text -SystemAdmin → sees ALL orgs -Others → sees only orgs they are a member of -``` - -### Team Visibility - -```text -SystemAdmin → sees ALL teams -OrgAdmin → sees all teams in orgs they admin + their own teams -Others → sees only teams they are a member of -``` - -### User Visibility - -```text -SystemAdmin → sees ALL users -Org members → sees users sharing at least one org -No orgs → sees only own profile -``` - -## Audit Visibility - -Audit log access uses the team→project→collection chain (not org-based): - -1. System collects all team IDs from the user's JWT -2. Finds all projects where `teams[]` overlaps with the user's teams -3. For each project, checks GET permission per collection -4. Builds a filter of allowed `{projectNamespace, collection}` pairs -5. Non-admin users **never** see `source.type: "system"` entries (login, reload, shutdown events) - -```text -SystemAdmin → sees ALL audit entries (including system events) -Team member → sees audit entries for collections they have GET access to -No teams → sees NO audit entries -``` - -## LLM Budget Cascade - -The LLM token budget system uses a two-level cascade: **User → Org**. - -### Budget Check Flow - -```text -1. Read user.llmBudgets[provider] ← REQUIRED (hard check) - └─ No budget → 429 Too Many Requests - └─ Period expired → 429 - └─ Tokens exhausted → 429 - -2. Read org.llmBudgets[provider] ← OPTIONAL (soft check) - └─ via user.primaryOrgId - └─ No primaryOrgId → skip org check - └─ Org has no budget for provider → no restriction - └─ Org budget expired → 429 - └─ Org tokens exhausted → 429 - -3. Execute LLM request - -4. Record usage: - └─ Deduct from user.llmBudgets - └─ Deduct from org.llmBudgets (if applicable) - └─ Write LLMUsageLog with userId AND orgId -``` - -### Configuration Example - -**Org with budget:** - -```json -{ - "name": "Acme Corp", - "llmBudgets": [{ - "provider": "openai", - "tokenBudget": 1000000, - "tokensUsed": 0, - "periodStart": "2026-01-01T00:00:00Z", - "periodEnd": "2026-04-01T00:00:00Z" - }] -} -``` - -**User with personal budget and primary org:** - -```json -{ - "username": "alice", - "primaryOrgId": "acme-org-id", - "llmBudgets": [{ - "provider": "openai", - "tokenBudget": 50000, - "tokensUsed": 0, - "periodStart": "2026-01-01T00:00:00Z", - "periodEnd": "2026-04-01T00:00:00Z" - }] -} -``` - -When Alice makes an LLM request: -- Her personal budget is checked first (hard check) -- Then Acme Corp's org budget is checked (soft check via `primaryOrgId`) -- Usage is deducted from both budgets -- The usage log entry includes both `userId` and `orgId` - -### Shared Budget Scenario - -Multiple users can share an org's budget: - -```text -Org "Acme": tokenBudget=1000, tokensUsed=985 -Alice (primaryOrgId=Acme): personal budget OK -Bob (primaryOrgId=Acme): personal budget OK - -Alice sends request → 15 tokens used → Org tokensUsed becomes 1000 -Bob sends request → 429 (org budget exhausted) -Alice sends request → 429 (org budget exhausted) -``` - -### Team Membership is Irrelevant for Billing - -LLM billing goes through `user.primaryOrgId`, not through team membership. A user in multiple orgs/teams is always billed to their primary org: - -```text -User in OrgA (primary) + OrgB → billing goes to OrgA only -``` - -## Cascade Consistency - -### Org Deletion Cascade - -When an org is deleted, the following cleanup occurs automatically: - -1. **Teams**: All teams with `orgId == deletedOrg` are deleted -2. **Projects**: `orgId` is unset, `teams[]` is cleared for all affected projects -3. **Users**: Org membership removed from `orgs[]`, plus: - - Teams belonging to the deleted org are removed from `teams[]` - - `primaryOrgId` is cleared if it referenced the deleted org - -### Team Deletion Cascade - -When a team is deleted: - -1. **Projects**: Team ID removed from `teams[]` -2. **Users**: Team membership removed from `teams[]` - -### User Org Removal Consistency - -When a user's `orgs[]` is modified and an org is removed: - -1. `primaryOrgId` is cleared if it referenced the removed org -2. Teams belonging to the removed org are automatically removed from `teams[]` - -## Access Control Summary - -### Org Endpoints - -| Action | SystemAdmin | OrgAdmin | Member | Unauthenticated | -|--------|-------------|----------|--------|-----------------| -| GET (list) | All | Own orgs | Own orgs | — | -| POST | ✅ | — | — | — | -| PUT | ✅ | — | — | — | -| DELETE | ✅ | — | — | — | - -### Team Endpoints - -| Action | SystemAdmin | OrgAdmin | Member | Unauthenticated | -|--------|-------------|----------|--------|-----------------| -| GET (list) | All | Org teams + own | Own teams | — | -| POST | ✅ | Own org teams | — | — | -| PUT | ✅ | Own org teams | — | — | -| DELETE | ✅ | Own org teams | — | — | - -### User Endpoints (org/team-related) - -| Action | SystemAdmin | OrgAdmin | TeamMember | -|--------|-------------|----------|------------| -| GET (list) | All | Shared-org users | Shared-org users | -| POST (create) | ✅ | ✅ (scoped) | — | -| PUT (modify orgs/teams) | ✅ | Own scope only | Own profile only | -| DELETE | ✅ | — | — | - -### Project Endpoints - -| Action | SystemAdmin | OrgAdmin | Member | -|--------|-------------|----------|--------| -| GET (list) | All | Org projects | Org projects | -| POST | ✅ | — | — | -| PUT (teams) | ✅ | ✅ (own org teams) | — | -| PUT (orgId, name, ...) | ✅ | — | — | -| DELETE | ✅ | — | — | - -## Field Protection - -When a non-SystemAdmin modifies a user, the following fields are protected: - -| Actor | Protected Fields | -|-------|-----------------| -| **OrgAdmin** | `permissions`, `llmBudgets` — cannot modify. `orgs`, `teams`, `primaryOrgId` — scoped to own orgs. | -| **TeamAdmin/Member** | `permissions`, `llmBudgets`, `orgs`, `teams`, `role`, `primaryOrgId` — all protected. Can only modify own profile's basic fields. | - -### OrgAdmin Scope Enforcement - -When an OrgAdmin modifies a user's `orgs[]` or `teams[]`: - -- **In-scope entries** (orgs the admin manages): can be added, removed, or role-changed -- **Out-of-scope entries** (other orgs): must remain unchanged — same number of entries, same roles -- Violation → `403 Forbidden` - -Example: OrgAdmin of OrgA edits a user who is in OrgA and OrgB: - -```json -// Before: -{ "orgs": [{"orgId": "A", "role": "member"}, {"orgId": "B", "role": "admin"}] } - -// Allowed (only OrgA changed): -{ "orgs": [{"orgId": "A", "role": "admin"}, {"orgId": "B", "role": "admin"}] } - -// Forbidden (OrgB modified): -{ "orgs": [{"orgId": "A", "role": "member"}, {"orgId": "B", "role": "member"}] } -``` diff --git a/src/content/docs/server/19-actions.md b/src/content/docs/server/19-actions.md deleted file mode 100644 index 0b14d45..0000000 --- a/src/content/docs/server/19-actions.md +++ /dev/null @@ -1,315 +0,0 @@ ---- -title: 'Actions' ---- - -# Actions - -## Übersicht - -Actions sind Hook-basierte Endpunkte ohne eine zugrunde liegende Datenbank-Collection. Sie eignen sich für Logik, die nicht an ein CRUD-Datenmodell gebunden ist, z.B. Kontaktformulare, Newsletter-Anmeldungen, Webhook-Empfänger oder benutzerdefinierte API-Endpunkte. - -Actions verwenden dasselbe Permission-, Hook-, CORS- und Audit-System wie Collections, haben aber **keine** Fields, Indexes, Uploads, Projections, Search oder Bulk-Operationen. - -## Konfiguration - -Actions werden im Projekt-Config unter dem `actions:`-Block definiert, analog zu `collections:`: - -```yaml -# config.yml -namespace: myproject - -collections: - - !include collections/events.yml - -actions: - - !include actions/contact-form.yml - - !include actions/newsletter-signup.yml -``` - -### Action-Config - -```yaml -# actions/contact-form.yml -name: contact-form - -meta: - label: { de: "Kontaktformular", en: "Contact Form" } - description: { de: "Verarbeitet Kontaktanfragen", en: "Processes contact requests" } - -permissions: - public: - methods: - post: true - user: - methods: - post: true - get: true - -hooks: - post: - bind: - type: javascript - file: hooks/contact-form/post_bind.js - validate: - type: javascript - file: hooks/contact-form/post_validate.js - handle: - type: javascript - file: hooks/contact-form/post_handle.js - return: - type: javascript - file: hooks/contact-form/post_return.js - get: - handle: - type: javascript - file: hooks/contact-form/get_handle.js - -cors: - allowOrigins: ["*"] -``` - -### Felder - -| Feld | Typ | Beschreibung | -|------|-----|-------------| -| `name` | string | Eindeutiger Name der Action (URL-Pfad-Segment) | -| `meta` | object | Metadaten wie Label, Description für Admin-UI | -| `permissions` | object | Identisch zu Collection-Permissions (public/user/role-basiert) | -| `hooks` | object | Hook-Definitionen pro HTTP-Methode und Step | -| `cors` | object | Optionale CORS-Konfiguration (überschreibt Projekt-CORS) | -| `audit` | object | Optionale Audit-Konfiguration (siehe Audit-Abschnitt) | - -## Admin-UI-Meta (tibi-admin-nova) - -Der Server liefert `meta` unverändert als JSON an den Admin aus (Passthrough — keine Validierung, kein Stripping, keine serverseitige Auswertung). Alles unter `meta.methods` ist reine UI-Konvention für Nova; der Hook entscheidet allein über das Verhalten, Nova erkennt es an der Response. Guards: Nova zeigt Sektion und Features nur bei vorhandener Meta (defensiv gegen alte Server ohne Meta). - -```yaml -# actions/import-snapshot.yml (Auszug) -meta: - label: { de: "Snapshot importieren", en: "Import Snapshot" } - muiIcon: file-upload-outline - description: { de: "Importiert eine CSV-Datei als Testlauf.", en: "Imports a CSV file as a test run." } - group: system - methods: - post: - data: - fields: - - name: file - type: file - validator: - required: true - accept: [".csv", "text/csv"] - maxFileSize: "2KB" - execution: - kind: sync # nur dokumentativ - output: - mapping: - summary: - fields: - - source: result.imported - type: number -``` - -Regeln: - -- `query.fields` / `data.fields`: Collection-Feldsprache (`type`, `label`, `validator`, `dependsOn` …). Nova rendert über dieselben Widgets und validiert clientseitig (`required`, `accept`, `maxFileSize`, Datei-Hinweise aus `accept`). Serverseitig gilt nichts automatisch — wer serverseitig härten will, schreibt einen `validate`-Hook in JS. -- Datei-Inputs immer inline als Base64/Data-URI im JSON-Body (Nova-Primärpfad, kein Multipart). -- SSE/Progress: Der Hook streamt bei Bedarf auf derselben Route (`context.response.header("Content-Type", "text/event-stream")`, `writeSSE`, `flush`); Nova erkennt `text/event-stream` per Content-Type und zeigt Fortschritt (`output.mapping.progress` mit `statusField`/`messageField`/`indexField`/`countField`, Progress-Text nur aus explizit konfigurierten Status-/Message-Feldern). `execution.kind` (`sync`/`sse`) ist nur dokumentativ, Transport entscheidet der Hook. -- `output.mapping.summary`/`details`: kompakte Ergebnisdarstellung über `source`-Pfade (dot-Notation, `name` nur Legacy-Fallback); ohne Mapping zeigt Nova Raw-JSON. -- Audit-Verlauf: Einträge liegen unter Collection `_actions/` und sind in Nova per Deep-Link (`/projects/:id/audit?collection=_actions/`) erreichbar. - -## Routing - -Actions sind unter folgendem Pfad erreichbar: - -``` -POST /api/v1/_/:project/_actions/:action -GET /api/v1/_/:project/_actions/:action -``` - -Der `_actions`-Prefix verhindert Namenskonflikte mit Collections. - -## Hook-Steps - -### POST - -| Step | Beschreibung | -|------|-------------| -| `bind` | Liest und transformiert den Request-Body. Kann Eingabedaten in `context` ablegen. | -| `validate` | Prüft die Eingabedaten. Wirft bei Fehlern eine Exception (HTTP 400). | -| `handle` | Führt die eigentliche Business-Logik aus. Kein DB-Roundtrip — alles passiert im Hook. | -| `return` | Transformiert die Ausgabe vor dem Senden der Response. | - -Ausführungsreihenfolge: `bind` → `validate` → `handle` → `return` - -### GET - -| Step | Beschreibung | -|------|-------------| -| `handle` | Erzeugt die Antwortdaten. | -| `return` | Transformiert die Ausgabe vor dem Senden der Response. | - -Ausführungsreihenfolge: `handle` → `return` - -## Hook-Context - -Action-Hooks haben Zugriff auf denselben Context wie Collection-Hooks, **ohne** die collection-/document-spezifischen Objekte: - -**Verfügbar:** -- `context.request()` — Request-Daten (body, query, headers, params) -- `context.data` — geparster JSON-Body (bei POST), identisch zum Collection-Verhalten -- `api.*` — Zugriff auf andere Collections via `api.find()`, `api.create()`, etc. -- `http`, `smtp`, `jwt`, `bcrypt`, `crypto`, `cache`, `ratelimit` — alle Standard-Packages - -**Nicht verfügbar:** -- `context.collection` — keine Collection vorhanden -- `context.document` — kein Dokument aus der DB - -## Response-Verhalten - -Die Response einer Action wird wie folgt bestimmt: - -1. Wenn der letzte Hook (z.B. `return` oder `handle`) ein Objekt mit `data`-Key zurückgibt, wird `data` als Response verwendet. -2. Wenn ein Objekt ohne `data`-Key zurückgegeben wird, wird das gesamte Objekt als Response verwendet. -3. Wenn kein Hook eine Response liefert, wird `{"ok": true}` zurückgegeben. - -### Beispiel: Response mit `data`-Key - -```javascript -// hooks/contact-form/post_handle.js -;(function () { - var body = JSON.parse(context.request().body()) - - // E-Mail senden via smtp Package - context.smtp.sendMail({ - to: "info@example.com", - subject: "Kontaktanfrage von " + body.name, - textBody: body.message - }) - - return { data: { sent: true, message: "Danke für Ihre Nachricht" } } -})() -``` - -Response: `{"sent": true, "message": "Danke für Ihre Nachricht"}` - -## Permissions - -Permissions funktionieren identisch zu Collections: - -```yaml -permissions: - public: - methods: - post: true # Jeder kann POST aufrufen - get: false # GET nur für eingeloggte User - user: - methods: - post: true - get: true - admin: - methods: - post: true - get: true -``` - -Rollen, Filter und `$subst`-Expressions werden genauso unterstützt wie bei Collections. - -## CORS - -CORS-Konfiguration auf Action-Ebene überschreibt die Projekt-CORS-Konfiguration: - -```yaml -cors: - allowOrigins: ["https://www.example.com"] - allowMethods: ["POST"] - allowHeaders: ["Content-Type", "X-Auth-Token"] -``` - -## Migration von Collection-Hacks - -**Vorher** (Collection ohne Fields als Action-Workaround): - -```yaml -collections: - - name: contact-form - strictFields: false - fields: [] - permissions: - public: - methods: - post: true - hooks: - post: - create: - type: javascript - file: hooks/contact-form/handle.js -``` - -**Nachher** (Native Action): - -```yaml -actions: - - name: contact-form - permissions: - public: - methods: - post: true - hooks: - post: - handle: - type: javascript - file: hooks/contact-form/handle.js -``` - -Vorteile: -- Keine unnötige DB-Collection -- Kein `fields: []`-Hack -- Eigener Hook-Step `handle` statt `create` -- Klarere Semantik und Trennung - -## Audit - -Actions unterstützen das gleiche Audit-System wie Collections. Audit-Logs werden unter der Collection `_actions/` gespeichert. - -### Konfiguration - -```yaml -# actions/contact-form.yml -name: contact-form - -audit: - enabled: true - # Optional: nur bestimmte Methoden auditieren (Standard: ["post", "get"]) - actions: - - post -``` - -Die Konfiguration ist identisch zu `AuditCollectionConfig` bei Collections: - -| Feld | Typ | Default | Beschreibung | -|------|-----|---------|-------------| -| `enabled` | bool | `false` | Audit für diese Action aktivieren | -| `actions` | []string | `["post", "get"]` | Welche HTTP-Methoden auditiert werden | - -> **Hinweis:** Audit muss zusätzlich global aktiviert sein (`AUDIT_ENABLED=true` oder `audit.enabled: true` in der Server-Config). - -### Audit-Einträge - -Audit-Einträge für Actions enthalten: - -| Feld | Beschreibung | -|------|-------------| -| `collection` | `_actions/` (z.B. `_actions/contact-form`) | -| `action` | HTTP-Methode (`post` oder `get`) | -| `snapshot` | JSON-safe Kopie der Hook-Daten (bei POST) | -| `userId`, `username`, `userRole` | Auth-Informationen (falls vorhanden) | -| `timestamp` | Zeitpunkt der Aktion | - -### Audit-Abfrage - -Audit-Logs für Actions können über den Standard-Audit-Endpunkt abgefragt werden: - -``` -GET /api/v1/_/:project/audit?filter={"collection":"_actions/contact-form"} -``` diff --git a/src/content/docs/server/20-referenced-read-safety.md b/src/content/docs/server/20-referenced-read-safety.md deleted file mode 100644 index e153f54..0000000 --- a/src/content/docs/server/20-referenced-read-safety.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: 'Referenced Read Safety' ---- - -# Referenced Read Safety - -## Problem - -Features that read from a second collection during a request can silently bypass parts of the target collection's normal GET visibility model if they only check collection-level `get` permission. - -Typical examples are: - -- lookup-like relation expansion -- reverse aggregates / derived counts and sums -- future facet, join, or enrichment features over referenced collections - -The dangerous gap is not only collection access itself, but also the field-level read model of the referenced collection. - -That includes at least: - -- permission-based pre-filters -- valid projections -- static hidden fields -- eval-based hidden fields -- read and return hooks -- user-supplied target filters that reference unreadable fields -- derived fields such as aggregate `valueField` - -## Project Invariant - -Any feature that reads or derives data from a referenced collection must go through the shared helper in [internal/controller/collection_reference_access.go](../internal/controller/collection_reference_access.go). - -Today that shared path is used by: - -- lookup target resolution -- aggregate target resolution - -The helper centralizes: - -- target collection existence and `get` permission checks -- effective pre-filter resolution -- projection validation / default projection selection for referenced reads -- hidden-field and projection-based field readability checks -- eval-hidden blocking for derived field access -- unreadable target field detection inside aggregate-style filters -- readable target join-field checks for lookup-style `foreignField` access - -## Practical Rule - -If a new feature touches a referenced collection, do not read target documents directly after `MethodIsAllowed` and `getCollectionPreFilter`. - -Instead: - -1. resolve referenced read access through the shared helper -2. validate any requested target field against that access model -3. validate any user-controlled target filter against readable fields -4. only then run target read hooks, DB reads, and return hooks - -For lookup specifically: - -- `foreignField` is a target read surface and must be readable for the caller -- static hidden fields may be pushed down into the referenced projection when that projection can express them safely -- eval-hidden stays a runtime sanitization concern and must not be treated as projection-only safe - -## Why This Matters - -Without this invariant, a feature can leak information even when the caller never receives the raw target document. - -Examples: - -- summing a hidden numeric field -- counting only documents where a hidden target field matches a secret condition -- using a target field blocked by projections as an aggregate input - -Those are still data leaks, even if the endpoint only returns `_aggregate` values. \ No newline at end of file diff --git a/src/content/docs/server/21-backup-restore.md b/src/content/docs/server/21-backup-restore.md deleted file mode 100644 index 4d2e760..0000000 --- a/src/content/docs/server/21-backup-restore.md +++ /dev/null @@ -1,534 +0,0 @@ ---- -title: 'Backup & Restore' ---- - -# Backup & Restore - -## Overview - -Die Backup & Restore-API ermöglicht das Exportieren eines kompletten Projekts (Collections + Media) -als ZIP-Datei sowie den Import auf dem gleichen oder einem anderen Server. - -**Nur Admin-User** können diese Endpunkte nutzen (Server-Token, Projekt-Token mit `allowBackup`, -oder JWT mit AdminRole). - ---- - -## Authentifizierung - -Dreistufiges Auth-Modell (identisch mit anderen Admin-Endpunkten wie `reload`): - -### 1. Server-Level Admin Token - -```yaml -api: - adminTokens: - - token: "backup-agent-token" - label: "ci-backup" - permissions: - - project.backup # Nur Backup/Restore - # - project.* # Alle Projekt-Operationen inkl. Backup -``` - -Gesendet via `X-Admin-Token` Header oder `Authorization: Bearer `. - -### 2. Projekt-Level Admin Token - -```yaml -# In der Projekt-Config (config.yml) -admin: - tokens: - - token: "ADMINTOKEN" - allowReload: true - allowRegenerateSearch: true - allowBackup: true # NEU: erlaubt Backup/Restore -``` - -Das Feld `allowBackup` steuert gezielt, ob dieser Token Backup/Restore verwenden darf. - -### 3. JWT Login mit AdminRole - -User mit `role: 0` (AdminRole) können Backup/Restore ausführen. - ---- - -## Endpunkte - -| Methode | Pfad | Beschreibung | -|--------|------|-------------| -| **GET** | `/_/:project/_/admin/backup` | Backup als ZIP herunterladen | -| **GET** | `/_/:project/_/admin/backup/download/:file` | Temp-ZIP aus SSE-Download abholen | -| **POST** | `/_/:project/_/admin/backup` | ZIP hochladen & analysieren → `restoreToken` (SSE via `?sse=1`) | -| **POST** | `/_/:project/_/admin/backup/:restoreToken` | Wiederherstellung ausführen | -| **DELETE** | `/_/:project/_/admin/backup/:restoreToken` | Temp-Daten manuell aufräumen | - ---- - -## 1. Backup Download - -```text -GET /_/:project/_/admin/backup?collections=contacts,pages&sse=1 -``` - -| Query-Param | Default | Beschreibung | -|-------------|---------|-------------| -| `collections` | _(alle)_ | Komma-separierte Collection-Namen (selektiver Export) | -| `sse` | `0` | `1` = Fortschritt als SSE-Stream, `0` = direktes ZIP | - -### Direkter Download (`sse=0`) - -Response: `Content-Type: application/zip` mit folgender Struktur: - -``` -backup--.zip -├── backup-info.json -├── collections/ -│ ├── .json # Array aller Dokumente -│ └── ... -└── media/ # Nur Dateien im konfigurierten uploadPath - └── / - └── / - └── / - └── -``` - -**Wichtig:** Es werden NUR Media-Dateien exportiert, die tatsächlich im konfigurierten -`uploadPath` der Collection existieren. Fehlende Dateien werden übersprungen und geloggt. - -Die Datei `backup-info.json` enthält Metadaten, die beim Import zur Schema-Validierung -verwendet werden: - -```json -{ - "type": "tibi-backup", - "version": 1, - "sourceNamespace": "mein-projekt", - "createdAt": "2026-06-24T12:00:00Z", - "collections": { - "contacts": { - "count": 42, - "fields": ["email", "firstname", "lastname", "phone"] - } - } -} -``` - -### SSE-Modus (`sse=1`) - -Response-Type: `text/event-stream` mit folgenden Events: - -``` -event: progress -data: {"phase":"init","totalCollections":3} - -event: result -data: {"phase":"done","downloadUrl":"/_/:project/_/admin/backup/download/","size":52428800,"expiresAt":"..."} -``` - -Der Client erhält eine temporäre Download-URL (TTL: 5 Minuten) und ruft die ZIP -anschließend separat ab: - -```bash -# Schritt 1: SSE-Download initiieren -curl -H "X-Admin-Token: $TOKEN" \ - "http://localhost:8080/api/v1/_/myproject/_/admin/backup?sse=1" - -# Schritt 2: ZIP vom temp-Link abholen -curl -H "X-Admin-Token: $TOKEN" \ - -o backup.zip \ - "http://localhost:8080/api/v1/_/myproject/_/admin/backup/download/" -``` - -Der Download-Link ist **One-Time-Use** — ein zweiter Request erhält 404. - ---- - -## 2. Backup Upload & Analyse (Phase 1) - -```text -POST /_/:project/_/admin/backup -Content-Type: multipart/form-data -``` - -**Body:** Feld `file` = backup.zip - -| Query-Param | Default | Beschreibung | -|-------------|---------|-------------| -| `sse` | `0` | `1` = Fortschritt via SSE streamen | - -### Direkter Modus (`sse=0`) - -**Response (200):** - -```json -{ - "restoreToken": "a1b2c3d4e5f6...", - "sourceNamespace": "quelle-projekt", - "createdAt": "2026-06-24T12:00:00Z", - "expiresAt": "2026-06-24T13:00:00Z", - "collections": { - "contacts": { - "documentCount": 42, - "mediaFileCount": 5, - "backupFields": ["email","firstname","lastname","phone"], - "targetFields": ["email","firstname","lastname","phone","department"], - "warnings": [ - "target has new fields not in backup: [department]" - ], - "exists": true - }, - "legacy_old": { - "documentCount": 7, - "mediaFileCount": 0, - "backupFields": ["data"], - "targetFields": null, - "warnings": ["collection \"legacy_old\" does not exist in target project"], - "exists": false - } - } -} -``` - -**In dieser Phase werden KEINE Daten geschrieben.** Das ZIP wird nur entpackt und analysiert. -Der `restoreToken` referenziert das Temp-Verzeichnis für die nächste Phase. - -### SSE-Modus (`sse=1`) - -Response-Type: `text/event-stream` mit folgenden Events: - -``` -event: progress -data: {"phase":"init","restoreToken":"a1b2c3d4e5f6..."} - -event: progress -data: {"phase":"save"} - -event: progress -data: {"phase":"extract"} - -event: progress -data: {"phase":"validate"} - -event: progress -data: {"phase":"analyze","totalCollections":2} - -event: result -data: {"phase":"done","restoreToken":"a1b2c3d4e5f6...","sourceNamespace":"quelle-projekt","createdAt":"2026-06-24T12:00:00Z","expiresAt":"2026-06-24T13:00:00Z","collections":{...}} -``` - -### Schema-Kompatibilitäts-Prüfung - -Die Prüfung ist **tolerant** — sie warnt, blockiert aber nicht: - -| Situation | Verhalten | -|-----------|-----------| -| Backup & Ziel identisch | Kein Warning | -| Backup hat Felder, Ziel nicht | Warning (MongoDB speichert sie trotzdem) | -| Ziel hat Felder, Backup nicht | Warning (Dokumente bekommen null/leer) | -| Collection existiert nicht im Ziel | `exists: false`, wird beim Restore geskippt | - ---- - -## 3. Restore ausführen (Phase 2) - -```text -POST /_/:project/_/admin/backup/:restoreToken?collections=contacts&merge=true&sse=1 -``` - -| Query-Param | Default | Beschreibung | -|-------------|---------|-------------| -| `collections` | _(alle)_ | Welche Collections aus dem Backup importieren | -| `merge` | `false` | `true` = Dokumente updaten, Orphans behalten; `false` = exakter 1:1-Abgleich (Orphans löschen) | -| `sse` | `0` | `1` = Fortschritt via SSE streamen | - -### Direkter Restore (`sse=0`) - -**Response (200):** - -```json -{ - "restoredCollections": ["contacts"], - "skippedCollections": ["legacy_old"], - "totalDocuments": 42, - "mediaFiles": 5 -} -``` - -### Restore-Modi - -**`merge=false` (Default) — Exakter 1:1-Abgleich:** -- Jedes Dokument aus dem Backup wird via atomarem Upsert (`ReplaceOne`) eingefügt oder ersetzt -- Nach dem Import werden alle Dokumente gelöscht, die vor dem Restore existierten, aber **nicht** im Backup enthalten sind (Orphans) -- Ergebnis: Die Collection ist eine exakte 1:1-Kopie des Backups — jedes Feld, jeder BSON-Typ -- Kein Zeitfenster, in dem die Collection leer ist (kein Pre-Delete mehr) -- Bricht der Import ab, sind bereits verarbeitete Docs korrekt, unverarbeitete unangetastet - -**`merge=true` — Überschreiben, Orphans behalten:** -- Existierende Dokumente mit gleicher `_id` werden **ersetzt** (atomares Upsert) -- Neue `_id`s werden **angelegt** -- Dokumente im Ziel, die nicht im Backup sind, bleiben **erhalten** -- Geeignet für partielle Updates / Migration einzelner Collections - -### SSE-Modus (`sse=1`) - -``` -event: progress -data: {"phase":"init"} - -event: progress -data: {"phase":"collection","collection":"contacts","action":"importing"} - -event: result -data: {"restoredCollections":["contacts"],"skippedCollections":[],"totalDocuments":42,"mediaFiles":5,"phase":"done"} -``` - ---- - -## 4. Temp-Daten aufräumen - -```text -DELETE /_/:project/_/admin/backup/:restoreToken -``` - -Explizites Cleanup des Temp-Verzeichnisses. Antwort: `{"message":"ok"}`. - -**Automatisches Cleanup:** Ein Background-Job (5-Minuten-Takt) löscht Temp-Verzeichnisse, -deren TTL abgelaufen ist: -- **Restore-Temp-Dirs**: 1 Stunde TTL -- **Download-ZIPs**: 5 Minuten TTL - ---- - -## Typischer Workflow - -### Szenario A: Projekt von Staging nach Production kopieren - -```bash -# 1. Auf Staging: Backup herunterladen -curl -H "X-Admin-Token: $STAGING_TOKEN" \ - -o staging-backup.zip \ - "https://staging.example.com/api/v1/_/myproject/_/admin/backup" - -# 2. Auf Production: Backup hochladen & analysieren -curl -H "X-Admin-Token: $PROD_TOKEN" \ - -F "file=@staging-backup.zip" \ - "https://prod.example.com/api/v1/_/myproject/_/admin/backup" -# → {"restoreToken":"abc...","collections":{...}} - -# 3. Prüfung der Analyse-Warnings, dann Restore ausführen -curl -H "X-Admin-Token: $PROD_TOKEN" \ - -X POST \ - "https://prod.example.com/api/v1/_/myproject/_/admin/backup/abc..." -# → {"restoredCollections":["contacts","pages"],"totalDocuments":84,"mediaFiles":12} -``` - -### Szenario B: Nur eine Collection aktualisieren (Selective Restore) - -```bash -# 1. Full Backup herunterladen (enthält alle Collections) -curl -H "X-Admin-Token: $TOKEN" -o full-backup.zip "..." - -# 2. Upload & Analyse -RESTORE_TOKEN=$(curl -H "X-Admin-Token: $TOKEN" -F "file=@full-backup.zip" "..." | jq -r .restoreToken) - -# 3. Nur "contacts" wiederherstellen -curl -H "X-Admin-Token: $TOKEN" -X POST \ - "http://.../_/myproject/_/admin/backup/$RESTORE_TOKEN?collections=contacts" -``` - -### Szenario C: Mit Merge updaten (bestehende Daten erhalten) - -```bash -# merge=true: Backup-Docs ersetzen bestehende, neue hinzufügen, -# aber Docs die nur im Ziel existieren bleiben erhalten -curl -H "X-Admin-Token: $TOKEN" -X POST \ - "http://.../_/myproject/_/admin/backup/$TOKEN?merge=true" -``` - -### Szenario D: Mit SSE-Fortschritt (große Projekte) - -```bash -# Download mit Fortschrittsanzeige -curl -H "X-Admin-Token: $TOKEN" -N \ - "http://.../_/myproject/_/admin/backup?sse=1" -# → event:progress + event:result mit downloadUrl - -# Restore mit Fortschrittsanzeige -curl -H "X-Admin-Token: $TOKEN" -N -X POST \ - "http://.../_/myproject/_/admin/backup/$TOKEN?sse=1" -``` - ---- - -## ZIP-Struktur (Referenz) - -``` -backup--.zip -├── backup-info.json # Metadaten + Schema-Fingerprint -├── collections/ -│ ├── contacts.json # Array aller Dokumente (mit _id) -│ └── pages.json -└── media/ # Nur Dateien im uploadPath - └── contacts/ - └── / - └── / - └── -``` - -Kein `project.json` oder `config.json` — das Projekt muss auf dem Zielsystem bereits existieren. - ---- - -## Media-Erkennung - -Beim Export werden **nur** Dateien berücksichtigt, die: -1. Über ein `file`- oder `file[]`-Feld im Dokument referenziert sind -2. **Tatsächlich** im konfigurierten `uploadPath` existieren (`os.Stat` → OK) -3. Innerhalb des `uploadPath` liegen (Path-Traversal-Schutz) - -Algorithmus: -``` -Für jedes Dokument einer Collection: - Für jedes Feld vom Typ "file" oder "file[]": - Relativer Pfad aus Feldwert extrahieren - Absoluter Pfad = / - os.Stat() → existiert → in ZIP aufnehmen - Existiert nicht → skip, Log-Warnung -``` - ---- - -## Sicherheit - -| Maßnahme | Beschreibung | -|----------|-------------| -| **Auth** | Server-Token, Projekt-Token, oder JWT AdminRole erforderlich | -| **Path Traversal** | `filepath.Clean` + `strings.HasPrefix` bei ZIP-Extraktion und Media-Kopie | -| **Temp-Files** | Automatischer Cleanup nach TTL, keine Akkumulation | -| **One-Time Download** | SSE-Download-Links sind nach einmaligem Abruf ungültig | -| **Media außerhalb uploadPath** | Pfade außerhalb des konfigurierten `uploadPath` werden ignoriert | - ---- - -## Fehlerbehandlung - -| HTTP-Status | Ursache | -|-------------|---------| -| **400** | Keine Datei hochgeladen, ungültiges ZIP, fehlende `backup-info.json`, falscher Typ/Version | -| **403** | Fehlende Berechtigung (`project.backup` oder `allowBackup`) | -| **404** | `restoreToken` nicht gefunden oder abgelaufen, Projekt nicht gefunden | -| **410** | Download-Link abgelaufen (nur bei `/download/:file`) | -| **500** | Interner Fehler (DB-Lesefehler, ZIP-Erstellung fehlgeschlagen) | - ---- - -## Temp-Datei-Management - -``` -Upload (Phase 1) → /tmp/tibi-backup// (TTL: 1h) -Restore (Phase 2) → importiert, dann gelöscht -DELETE /:restoreToken → sofort gelöscht -TTL-Ablauf → Background-Cleanup (alle 5 Min) - -SSE Download → /tmp/tibi-backup-download/.zip (TTL: 5 Min) -GET /download/:file → ausgeliefert + sofort gelöscht -``` - ---- - -## Konfiguration - -### Server-Level Token (config.yml) - -```yaml -api: - adminTokens: - - token: "backup-token" - label: "backup-agent" - permissions: - - project.backup -``` - -`project.*` (Wildcard) inkludiert automatisch `project.backup`. - -### Projekt-Level Token - -```yaml -admin: - tokens: - - token: "PROJEKT-TOKEN" - allowBackup: true -``` - -### Keine Projekt-Config-Änderung nötig - -Der Backup/Restore-Vorgang benötigt keine Änderungen an der Projekt-Config. -Collections, Felder, uploadPath etc. werden aus der bestehenden Konfiguration gelesen. - ---- - -## Implementierung - -### Dateien - -| Datei | Inhalt | -|-------|--------| -| `internal/controller/admin_backup.go` | Alle Handler (Download, Analyze, Restore, Cleanup, SSE, Temp-Stores) | -| `internal/controller/admin_backup_auth_test.go` | 12 Auth-Tests | -| `internal/controller/admin_backup_analyze_test.go` | 8 Analyze- & Download-Tests | -| `internal/controller/admin_backup_restore_test.go` | 9 Restore-Tests | -| `internal/controller/admin_backup_integration_test.go` | 3 SSE/E2E-Tests | -| `internal/controller/setup.go` | Routen-Registrierung | -| `internal/models/project.go` | `AdminToken.AllowBackup` Feld | -| `cmd/server/main.go` | `StartBackupCleanupJob()` Aufruf | - -### Test-Statistik - -**33 Tests** (alle grün): -- 12 Auth-Tests (Server-Token, Projekt-Token, JWT, Berechtigungen) -- 6 Analyze-Tests (ZIP-Validierung, Schema-Diffs, unbekannte Collections) -- 2 Download-Tests (direkt, selektiv) -- 9 Restore-Tests (Import, Merge, Selective, Media-Copy, Cleanup) -- 3 SSE/E2E-Tests (SSE-Download, DownloadFile, SSE-Restore) -- 1 E2E-Test (skipped — Mock-Projekt-API-Limitierung) - -Keine Regression auf bestehenden Admin-Tests (`TestAdminReload`, `TestAdminAuthReload` etc.). - ---- - -## Betrieb: Speicherverhalten seit Streaming-Umbau (2026-10) - -Erstellung und Wiederherstellung von Snapshots und Backups laufen in -**O(Batch)** statt O(Collection): Keyset-Pagination (`_id $gt`, Sort `_id`), -Datei-/ZIP-Streaming, sequenziell pro Collection. RAM ist keine Grenze mehr — -auch 10-GB-Collections (z. B. Outdoorworks: `productVariant` 91k × ~117 KB mit -`_search`-Bloat) laufen mit konstantem Heap. - -### Batch-Konstanten (`internal/controller/snapshot_stream.go`) - -| Konstante | Wert | Wirkung | -|---|---|---| -| `SnapshotBatchDocs` | 500 | Docs pro DB-Batch und Restore-Batch | -| `SnapshotBatchBytes` | 8 MiB | reserviert (Byte-Cap, greift pro großem Einzeldoc) | -| `SnapshotBufSize` | 64 KiB | `bufio`-Puffer für Snapshot-Dateien | -| `OrphanMapThreshold` | 200.000 | darunter In-RAM-Orphan-Set, darüber Temp-Collection in Ziel-DB | -| `orphanChunkSize` | 500 | `$in`-Chunks für Orphan-Deletes | - -### Empfehlungen - -- `GOMEMLIMIT` setzen (z. B. `GOMEMLIMIT=512MiB` im Container): begrenzt GC-Druck, - Streaming braucht es nicht — aber es schützt vor Nachbar-Last. -- Jeder Snapshot-Run loggt `snapshot : done documents=… heapGrowthMB=…` - (`heapGrowthMB` = Heap-Wachstum über den Lauf, kein Peak; Peak ist per - Konstruktion batch-begrenzt). -- Retention `maxCount` behält die **neuesten** Einträge (Bugfix 2026-10-15: - zuvor wurden fälschlich die neuesten gelöscht). - -### Nachweise (Tests) - -- `TestExportStreaming_PeakScalesFlat` / `TestStreamRestore_PeakScalesFlat`: - 10× Docs ⇒ kein 10× Peak (Kontrollmessung Alt-Pfad liegt ein Mehrfaches darüber). -- E2E Roundtrips (Mock + Real-Mongo): Snapshot NonMerge/Merge, Backup - Restore-Matrix, Download-Pipelines. -- OOM-Fixture: `testdata/outdoorworks-oom/` (echte `_search`-Form, Stats, - `dump-full.sh` für den 8-GB-Voll-Dump nach `/WM_Dev/.cortex/mcp-tmp/`). diff --git a/src/content/docs/server/22-personal-access-tokens.md b/src/content/docs/server/22-personal-access-tokens.md deleted file mode 100644 index 0c9478d..0000000 --- a/src/content/docs/server/22-personal-access-tokens.md +++ /dev/null @@ -1,295 +0,0 @@ ---- -title: '22. Personal Access Tokens' ---- - -# 22. Personal Access Tokens - -Self-managed API tokens for users. Every user can create, list, update and delete -their own tokens. A token can be restricted to a subset of the owner's rights and -is **always** checked live against the current rights of its owner. - -- [Token format & storage](#token-format--storage) -- [Quick start](#quick-start) -- [Scope model](#scope-model) -- [Scope reference](#scope-reference) -- [Which scopes a user may grant](#which-scopes-a-user-may-grant) -- [Endpoint → scope mapping](#endpoint--scope-mapping) -- [Management API](#management-api) -- [Security properties](#security-properties) -- [Status codes](#status-codes) - -Personal Access Tokens are a core feature and always enabled. - -## Token format & storage - -```text -tibi_pat_<24-hex objectId>_ -``` - -- Only `sha256(secret)` is stored (`tokenHash`); the plaintext exists exactly once, - in the create response. -- `tokenPrefix` (`tibi_pat_` + 4 secret chars, 13 chars total) is stored for - display and audit. -- Lookup uses the embedded object id → one indexed `_id` read. -- The secret is compared in constant time. -- No expiry by default; `expiresAt` is optional (`expiresAt` must be in the - future when minting). - -A PAT is sent via **`X-Auth-Token`** (like the JWT). `Authorization: Bearer` is -reserved for project-specific authentication and is *not* accepted for PATs. - -## Quick start - -```bash -# 1) create (requires a user login session, not a PAT) -curl -X POST https://host/api/v1/token \ - -H "X-Auth-Token: " \ - -H "Content-Type: application/json" \ - -d '{"name":"CI Deploy","scopes":["namespace.website.articles.read"]}' - -# 2) use the returned plaintext token (shown exactly once) -curl https://host/api/v1/_/website/articles \ - -H "X-Auth-Token: tibi_pat_..." - -# 3) revoke (immediately effective) -curl -X DELETE https://host/api/v1/token/ -H "X-Auth-Token: " -``` - -## Scope model - -A scope is a dotted permission string. Three rules define the effective access of -a PAT request: - -1. **Token ⊆ owner.** A scope can only be granted if the owner currently holds - that permission. The catalog is computed live from the database at - create/update time. -2. **Scope ∩ owner (AND).** At request time the action must be allowed by the - **token scope** *and* by the **owner's current rights**. Revoking a user right - (role, org, team, `permissions[]`) restricts the token immediately. -3. **Same syntax as admin tokens.** Dotted segments, `*` wildcard per segment, - and shorthands where fewer segments imply all descendants. Because of rule 2, - even a very broad scope (up to `*`) can never grant more than the owner - currently holds. - -**Separate credentials authorize independently.** A valid **server/project admin -token** (`X-Admin-Token` or `Authorization: Bearer`) or a **collection config -token** (`name:value` header/query) authorizes on its own; passing an additional -narrow PAT does not restrict such a request. PAT scopes only restrict requests -that are authenticated *as a user* by the PAT. - -Additional semantics: - -- **Public baseline.** Data that is publicly readable stays readable with any PAT - (even with no scopes) — a token never adds or removes anonymous access. -- **Empty scope list** is allowed and yields a token that can only reach public - data. `*` yields the full (but still owner-capped) access. -- **Shorthands** are prefixes and never grant more than the owner has. - -### Shorthand resolution - -| Scope written | Matches (at most the owner's rights) | -|---------------|--------------------------------------| -| `*` | everything the owner may do | -| `namespace` | all collections in all namespaces | -| `namespace.` | every collection/action in `` the owner may use | -| `namespace..` | every action on `` the owner may use | -| `namespace...` | exactly that action | -| `user` / `project` / `team` / `org` | all actions of that system domain | -| `user.read` / `project.reload` / `audit.read` / `llm.chat` | exactly that action | -| `namespace.*..read` / `user.*` | wildcard matches any single segment | - -Because of rule 2, a shorthand or wildcard can never exceed the owner's rights — -it only avoids enumerating them. `*` is the convenient "everything this user may -do" scope; a request is still rejected as soon as the owner lacks the right. - -## Scope reference - -### Namespace (project collection) scopes - -Namespace = the project's `namespace` (see project config). - -| Scope | Grants | -|-------|--------| -| `namespace...read` | `GET /_//` (list/one), `_count`, `_facet`, `_queue`, file reads, lookup/aggregate reads of `` (as target), audit entries of `` | -| `namespace...create` | `POST /_//` (single and bulk) | -| `namespace...update` | `PUT /_///:id` (single and bulk) | -| `namespace...delete` | `DELETE /_///:id` (single and bulk; bulk additionally needs the owner's bulk right) | -| `namespace..` | all of the above the owner holds | -| `namespace.` | all of the above for every collection in `` | -| `namespace` | all of the above for every collection the owner may use | -| `*` | everything below and the system domains the owner may use | - -### Action endpoint scopes - -Actions (`/_//_actions/`) have their own scopes. - -| Scope | Grants | -|-------|--------| -| `namespace.._actions..read` | `GET /_//_actions/` | -| `namespace.._actions..create` | `POST /_//_actions/` | - -### System domain scopes - -| Scope | Grants | -|-------|--------| -| `user` | all user actions below | -| `user.read` | `GET /api/v1/user`, `GET /api/v1/user/:id` (owner-visibility applied) | -| `user.create` | `POST /api/v1/user` | -| `user.update` | `PUT /api/v1/user/:id` | -| `user.delete` | `DELETE /api/v1/user/:id` | -| `project.read` | `GET /api/v1/project`, `GET /api/v1/project/:id` | -| `project.create` | `POST /api/v1/project` | -| `project.update` | `PUT /api/v1/project/:id` | -| `project.delete` | `DELETE /api/v1/project/:id` | -| `project.reload` | `POST /_//_/admin/reload` | -| `project.backup` | `POST/GET/DELETE /_//_/admin/backup…` | -| `project.regenerate-search` | `POST /_//_/admin/regenerate-search` | -| `project.snapshot` | `/_//_/admin/snapshots…` (list/detail/restore/download/prune) | -| `team` / `org` | all actions below | -| `team.read` / `team.create` / `team.update` / `team.delete` | `/api/v1/team…` | -| `org.read` / `org.create` / `org.update` / `org.delete` | `/api/v1/org…` | -| `audit.read` | `GET /api/v1/audit` (see below) | -| `llm.chat` | `POST /api/v1/llm/chat` | - -**Audit is special:** a PAT needs `audit.read` **and** the read scope of a -collection (`namespace...read`) to see that collection's audit entries. -`audit.read` alone does not expose collection snapshots. Non-admin users never see -system events (login/reload/shutdown) — this is unchanged by tokens. - -**Not scope-relevant (public by design):** `GET /_//_/assets/…`, `_meta`, -`_openapi/:project`, `/version`, `GET /api/v1/llm/providers`. - -## Which scopes a user may grant - -`GET /api/v1/token/_scopes` returns the exact catalog for the calling user. In -general: - -| Owner role | System scopes available | -|------------|-------------------------| -| SystemAdmin (`role: 0`) | all of the above, including `project.reload/backup/regenerate-search/snapshot` | -| OrgAdmin (`orgs[].role: "admin"`) | `user.read/create/update`, `team.read/create/update/delete`, `project.read/update`, `org.read`, `audit.read`, `llm.chat` | -| User (`role: 1`) | `user.read/update`, `project.read`, `team.read`, `org.read`, `audit.read`, `llm.chat` | - -Collection and action scopes are additionally limited to the projects/collections -the owner can reach (org visibility + team permissions). Public-only rights are -never part of the catalog. - -## Endpoint → scope mapping - -| Endpoint | Required scope | -|----------|----------------| -| `GET /_//` (+ `_count`, `_facet`, `_queue`) | `namespace...read` | -| `POST /_//` | `namespace...create` | -| `PUT /_///:id` | `namespace...update` | -| `DELETE /_///:id` | `namespace...delete` | -| `GET /_///:id/:field/*path` (file) | `namespace...read` (public collections: none) | -| `GET /_//?lookup=…:` | `namespace...read` **and** `namespace...read` (public target: none) | -| `GET /_//?aggregate=…:` | `namespace...read` **and** `namespace...read` (public target: none) | -| `GET /api/v1/…?lookup=…:` (system lookup) | `.read` **and** `.read`; missing target scope → target is silently not embedded | -| `GET/POST /_//_actions/` | `namespace.._actions..read` / `.create` | -| `/api/v1/user…` | `user.` | -| `/api/v1/project…` | `project.` | -| `/_//_/admin/reload` etc. | `project.reload` / `.backup` / `.regenerate-search` / `.snapshot` | -| `/api/v1/team…`, `/api/v1/org…` | `team.`, `org.` | -| `GET /api/v1/audit` | `audit.read` (+ per-collection read scope) | -| `POST /api/v1/llm/chat` | `llm.chat` | - -`GET /api/v1/token*` itself is **not** scope-controllable: token management -requires a user login session (a PAT gets `403`). - -## Management API - -All endpoints operate on the caller's own tokens; system admins can see and -manage all. - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/api/v1/token` | list own tokens (never contains secret/hash) | -| `POST` | `/api/v1/token` | create → returns the plaintext token **once** | -| `GET` | `/api/v1/token/_scopes` | available scopes for the current user | -| `PUT` | `/api/v1/token/:id` | update name/scopes/expiry (re-validated against the owner catalog) | -| `DELETE` | `/api/v1/token/:id` | delete (revoke); effective immediately | - -Create response (the `token` field appears only here): - -```json -{ - "id": "60a1b2c3d4e5f6a7b8c9d0e1", - "name": "CI Deploy", - "tokenPrefix": "tibi_pat_ab12", - "scopes": ["namespace.website.articles.read"], - "expiresAt": null, - "token": "tibi_pat_60a1b2c3d4e5f6a7b8c9d0e1_..." -} -``` - -## Security properties - -- **Never more than the owner:** scopes are validated against the owner's live - rights when minted; every request re-checks the owner's current rights - (scope ∩ owner). -- **Immediate revocation:** deleting a token invalidates it on the next request - (no validity cache); user permission/role/org/team changes take effect - immediately. -- **No secret leakage:** only the hash is persisted; `tokenHash` is never - serialized; audit entries record `authMethod="access-token"`, `tokenLabel` - (name) and `tokenPrefix`, never the secret or hash. -- **Ownership:** `PUT`/`DELETE` are restricted to the owner (or system admin). -- **Session-only management:** `/api/v1/token*` (including `/_scopes`) requires a - user login session; a PAT cannot mint/manage tokens or impersonate users. -- **Impersonation:** tokens cannot be created or managed while impersonating. -- **User deletion:** all tokens of the user are removed (cascade). - -## UI integration notes (tibi-admin-nova) - -- **Session required:** management calls use the normal login session - (`X-Auth-Token: `). PATs are rejected on `/api/v1/token*` with `403`. -- **Scope picker:** `GET /api/v1/token/_scopes` returns `string[]` — the exact - scopes the current user may grant (concrete collection/action scopes, namespace - and collection shorthands, system actions). Offer them as checkboxes plus a - "select all" (= everything the user may do). -- **Power input:** arbitrary admin-token-style scopes are allowed (`*`, `user`, - `namespace.website`). The server validates them and answers `400` for invalid - syntax or scopes unrelated to the user's rights. -- **Create:** `POST /api/v1/token` with `{"name": "...", "scopes": [...], - "expiresAt": null}`. The response contains the plaintext `token` **exactly - once** — show a copy-once dialog and never persist it in the frontend state - longer than needed. -- **List:** `GET /api/v1/token` → - `[{id, name, tokenPrefix, scopes, expiresAt, lastUsedAt, insertTime, updateTime}]`. - Never contains `token` or `tokenHash`. -- **Update:** `PUT /api/v1/token/:id` with any of `name`, `scopes`, `expiresAt` - (re-validated; `expiresAt` must be in the future). -- **Revoke:** `DELETE /api/v1/token/:id` — effective immediately. -- **Errors:** `403` (no/insufficient session, foreign token, missing scope), - `400` (invalid syntax, scope not permitted, past `expiresAt`), `404` (unknown - token id). Show the `error` string. -- **Display list entries with** `name`, `tokenPrefix` and `lastUsedAt`; the - plaintext is never shown again. - -## Known limitations - -These are accepted, documented residual topics (deliberately not implemented): - -- **No rate limit / cap on token creation.** Rate limiting (`ratelimit`) applies - to `POST /api/v1/login` only. An authenticated session can mint an arbitrary - number of tokens; every create/delete is audited, but there is no per-user cap. - If needed, add a limiter to `/api/v1/token` and/or a max-tokens-per-user config. -- **Invalid-token requests cost 1–2 indexed DB reads** (token + owner). Repeated - requests with arbitrary `tibi_pat_…` values are not throttled, which is a mild - amplification vector. Mitigation would be auth-failure rate limiting. - -Neither is a privilege escalation: a token never exceeds its owner and never -exposes secrets. - -## Status codes - -| Situation | Status | -|-----------|--------| -| Missing/invalid/expired/deleted token | `403` | -| Token valid, scope missing | `403` | -| Scope present, owner right missing | `403` | -| `POST`/`PUT` with a syntactically invalid scope (bad segments, unknown domain) | `400` | -| `POST`/`PUT` with a scope not related to any owner right | `400` | -| `expiresAt` in the past | `400` | -| Foreign token `PUT`/`DELETE` | `403` (owner or system admin only) | diff --git a/src/content/docs/server/23-mcp.md b/src/content/docs/server/23-mcp.md deleted file mode 100644 index 3391fce..0000000 --- a/src/content/docs/server/23-mcp.md +++ /dev/null @@ -1,156 +0,0 @@ ---- -title: '23. MCP-Server' ---- - -# 23. MCP-Server - -Tibi-server ist zusätzlich ein MCP-Server (Streamable HTTP, JSON-RPC 2.0) für -Agenten, die Websites befüllen. Alle Tools laufen mit denselben Checks wie REST. - -- [Scope-Modell](22-personal-access-tokens.md) · [Auth](05-authentication.md) · - [Collections](04-collections.md) · [Actions](19-actions.md) - -## Transport & Auth - -``` -POST /api/v1/mcp # JSON-RPC: initialize, tools/list, tools/call -GET /api/v1/mcp # Capability-Probe (PAT nötig) -``` - -- Auth **nur** per PAT: `X-Auth-Token: tibi_pat_...` -- JWT, `X-Admin-Token`, `Authorization: Bearer` (non-PAT) → `403`. -- Deaktiviert (`mcp.enabled: false`) → `404`. - -```bash -# Tools listen -curl -X POST https://host/api/v1/mcp \ - -H "X-Auth-Token: tibi_pat_..." -H 'Content-Type: application/json' \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' - -# Tool aufrufen -curl -X POST https://host/api/v1/mcp \ - -H "X-Auth-Token: tibi_pat_..." -H 'Content-Type: application/json' \ - -d '{"jsonrpc":"2.0","id":2,"method":"tools/call", - "params":{"name":"collections_list","arguments":{"namespace":"website"}}}}' -``` - -JSON-RPC-Fehlercodes: `-32601` unbekanntes Tool, `-32004` Resource nicht gefunden, -`-32003` Scope/Permission (`403`), `-32602` Validierung (`400`), `-32603` intern (`500`), -`-32700` Parse-Error. - -## Konfiguration (`config.yml`) - -```yaml -mcp: - enabled: true - route: "/api/v1/mcp" - maxResultItems: 100 - allowAdminActions: true # reload/backup/snapshot nur wenn Token-Scope es erlaubt -``` - -Env: `MCP_ENABLED`, `MCP_ROUTE`, `MCP_MAX_RESULT_ITEMS`, `MCP_ALLOW_ADMIN_ACTIONS`. -Kein `allowWrite`-Flag (nur PAT-Scopes), kein `llm.chat` via MCP. - -## Tools (Hybrid, 36) - -Discovery: `projects_list`, `project_get`, `collections_list` (braucht `project.read`), -`collection_describe` (braucht `project.read`; rechtegefilterte Wahrheit: Felder, Validatoren, -`yourPermissions`, Pre-Filter, Projektionen, `meta.agent` ohne Secrets, Perm-Keys maskiert), -`actions_list`, `action_describe`, `jobs_list` (lesend, kein Trigger), -`openapi_get`, `system_meta_get`, `agent_guide_get` (kuratiert aus `meta.agent`, inkl. `yourPermissions` bei Collection), -`assets_list`. `asset_read` ist REST-only (`GET /_/:project/_/assets/...` nutzen). - -CRUD: `documents_list` (filter/sort/limit/offset/select/projection/lookup/aggregate/q -plus `extra`-Objekt für `meta.query`-Parameter, s. unten), -`document_get`, `documents_count`, `documents_facet`, `queue_status`, -`document_create` (single + `items[]`-Bulk, Dateien als base64 via `CollectionPost`), -`document_update` (single + `filter`-Bulk), `document_delete`, -`action_call` (nur get/post). - -System: `users_list`, `user_get`, `orgs_list`, `org_get`, `teams_list`, `team_get` -(je mit `ValidateFilter`-Sichtbarkeit), `audit_query` -(`audit.read` + Collection-`read`; System-Events für Non-Admin unsichtbar). - -REST-only in dieser Phase (klare `-32602`-Meldung, kein Silent-Fail): -`user_create/update/delete`, `project_reload`, `regenerate-search`, -`backup_*`, `snapshot_*`, `file_read`, `asset_read` → REST-Endpunkte nutzen. -Nie via MCP: `token`-Management, `login`/`impersonate`, `shutdown`, `llm.chat`. - -## Endpoint → Scope (Auszug) - -| MCP-Tool | Scope | -|---|---| -| `projects_list`/`project_get` | `project.read` | -| `collections_list` | `project.read` + je Collection `namespace...read` | -| `collection_describe` | `project.read` + `namespace...read` | -| `documents_*`, `queue_status` | `namespace...read` | -| `document_create` | `namespace...create` | -| `document_update`/`delete` (bulk) | `.update`/`.delete` (+ Owner-Bulk-Recht) | -| `action_call` | `namespace.._actions..read`/`.create` | -| `users_*`/`orgs_*`/`teams_*` | `user/org/team.read` (+ Write-Scopes für REST-only) | -| `audit_query` | `audit.read` (+ Collection-`read`) | - -Dazu gelten Pre-Filter, Projektionen (`select` nur einengend), Bulk-Flags, -Lookup/Aggregate-Target-Scopes, Validatoren, Hooks (Hooks sind methoden-keyed: -jedes Tool nutzt seine REST-Methode, damit `get`/`put`/`delete`-Hooks laufen) -und Audit-Einträge mit `tokenLabel`/`tokenPrefix` (Client-IP vom Aufrufer) — -identisch zu REST, weil dieselben Controller laufen. MCP-Einträge tragen -zusätzlich `channel: "mcp"` (REST lässt das Feld leer), abfragbar per -`filter={"channel":"mcp"}`. - -## Agent-Anleitung (`meta.agent`, YAML-Konvention) - -`meta` ist free-form; der Unterschlüssel `agent` ist die kuratierte Anleitung. -Erlaubte Keys: `description`, `purpose`, `instructions`, `frontend`, `workflows`, -`hints`, `fieldHints`, `examples`. Secrets/Token/Pfade/Hooks werden nie -durchgereicht (Größe gecappt). - -```yaml -# config.yml -meta: - agent: - description: { de: "Stadtportal", en: "City portal" } - frontend: { routes: "/:slug, /news/:id" } - workflows: ["Artikel anlegen → publizieren → prüfen"] - hints: ["slug im kebab-case"] - -# collections/articles.yml -meta: - agent: - purpose: { de: "News-Artikel für Startseite" } - instructions: "Teaser max 160 Zeichen, publishDate immer setzen." - fieldHints: { teaser: "max 160 Zeichen" } - examples: [{ title: "...", published: true }] -``` - -Idempotenz: `filter`-vor-`create` + `unique`-Indexe; `409` = vorhanden; -stabile Caller-`_id` bei Retry. - -## Zusätzliche Query-Parameter (`meta.query`) - -Hooks werten oft Query-Parameter aus (`ctx.request().query(name)`). Via MCP sind -nur freigegebene Argumente erlaubt — Collections deklarieren weitere Parameter -in `meta.query` (Schema in `tibi-types/schemas/config/collection-meta.schema.json`): - -```yaml -# collections/articles.yml -meta: - query: - hook: { type: string, enum: [fakedata, mail], description: { de: "Test-Dispatcher" } } - view: { type: string, maxLength: 32 } -``` - -- Typen: `string`, `number`, `boolean`; dazu `enum` und `maxLength`. -- MCP: als `extra`-Objekt an `documents_list`, `document_get`, `documents_count`, - `documents_facet` übergeben — validiert, dann als Query gesetzt. Unbekannte oder - falsch typisierte Keys → `-32602`. Sichtbar in `collection_describe` unter `query`. -- Niemals freigegeben (auch nicht konfigurierbar): Credential-Kanäle - (Permission-Token-Namen, `x-auth-token`, `x-admin-token`, `authorization`) und - Core-Keys (`filter`, `sort`, `limit`, `select`, `lookup`, …). -- REST bleibt unverändert offen; das Schema dient dort als Dokumentation - (`_meta`/OpenAPI) für Agenten. Details: [Collections](04-collections.md#metaquery). - -## Beispiele - -Siehe `examples/mcp/` (curl: tools/list, describe, Befüllungs-Workflow). -`test.http` enthält MCP-Snippets. diff --git a/src/content/docs/server/24-ephemeral-rest-tokens.md b/src/content/docs/server/24-ephemeral-rest-tokens.md deleted file mode 100644 index 1cc567a..0000000 --- a/src/content/docs/server/24-ephemeral-rest-tokens.md +++ /dev/null @@ -1,75 +0,0 @@ ---- -title: '24. Ephemere REST-Tokens via MCP (PLAN)' ---- - -# 24. Ephemere REST-Tokens via MCP (PLAN) - -Status: Plan — noch nicht implementiert. Ziel: harness-neutrale Kurzzeit-Credentials, -damit Shell/`curl` alles tun kann, was MCP kann, ohne Base64 in den LLM-Kontext zu ziehen. - -## Ausgangslage - -- MCP-Uploads (`document_create`/`document_update`, `type:file` als - `{src: "data:;base64,...", path: "name.jpg"}`) funktionieren, landen aber bei - reinen Tool-Harness (kein JS-Runtime) als ~12k Tokens pro Bild im Prompt. - `documents_list` ohne `select` spült `file.data` ebenso zurück. -- Mit JS-Runtime (OpenCode `execute`: `fetch.bytes()` → `mcp_call`) bleibt Base64 im - Runtime-Speicher, zurück kommt nur `{id, file:{src,path,type}}`. Das hilft nur dort. -- Upload-Tickets (Presigned-PUT, S3, CORS, Quota) wären ein eigenes Projekt. - -## Idee - -Generische Kurzzeit-Token für alles, was MCP kann, via REST: MCP stellt aus, -Shell verbraucht. Binär/Base64 baut die Shell aus Dateien (`base64`, `jq`, `curl`), -im Prompt stehen nur Token (32 Zeichen) + Pfade. - -```bash -# 1) via MCP ausstellen (Scope: ns.collection.method, TTL z. B. 300s) -# -> zurück: tibi_eph_... (einmalig im Klartext) -# 2) in der Shell verbrauchen, ohne LLM-Bloat -curl https://host/api/v1/_/website/articles \ - -H "X-Auth-Token: tibi_eph_..." \ - -H 'Content-Type: application/json' \ - -d @/WM_Dev/.cortex/mcp-tmp/payload.json -curl https://host/api/v1/_/website/medialib//file/name.jpg \ - -H "X-Auth-Token: tibi_eph_..." -``` - -## Design (angelehnt an 22-PAT) - -- Format/Storage wie PAT wiederverwenden (`tokenHash` statt Klartext, `tokenPrefix` - für Anzeige/Audit, Lookup über eingebettete Objekt-ID, Constant-Time-Vergleich), - plus Pflichtfelder: `expiresAt` (max. 1h, Default 5 Min), `ephemeral:true`, - `parentTokenId` (Audit-Verkettung), enger Scope (z. B. genau eine Collection+Methode). -- Scope-Regeln wie PAT übernehmen: **Token ⊆ Owner** bei Ausstellung (live aus DB), - **Scope ∩ Owner** bei jedem Request (Owner-Revoke wirkt sofort). Kein neues - Rechtesystem, keine neuen Header: weiter `X-Auth-Token` wie PAT. -- Transport: neues MCP-Tool `token_issue_ephemeral` (braucht die zu delegierende - Scope-Menge selbst) + optional `token_revoke_ephemeral`. Kein `login`/`impersonate` - via MCP (bleibt verboten wie in 23-MCP). REST-Seite unverändert außer TTL-Check. -- Missbrauchsschutz: kurze TTL + vorhandener `expiresAt`-Pfad, Audit wie PAT, - Rate-Limit auf Ausstellung, kein Refresh (neu ausstellen statt verlängern), - keine Scope-Erweiterung über Owner hinaus. - -## Nicht-Ziele - -- Kein Presigned-S3, kein Multipart-Spezialfall, keine Änderung an - `handleFileInJSON` (`src`/`path`-Vertrag aus 08-File-Upload bleibt). -- Kein Ersatz für `clear_cache`-Hooks oder `filter_public`-Vertrag. -- Keine langlebigen CI-Token (dafür weiter normale PAT mit `expiresAt`). - -## Umsetzungsskizze - -1. `token_issue_ephemeral` im MCP (Scope-Check gegen Caller-PAT, TTL clamp 60s–3600s). -2. Token-Dokuments mit `expiresAt` + TTL-Index/Cleanup (prüfen, ob PAT-Cleanup das - schon abdeckt, sonst Job ergänzen). -3. Doku in 22-PAT (Ephemeral-Absatz) + 23-MCP (Tool-Zeile) + Beispiel-Shell-Snippet. -4. Tests: Ablauf nach TTL, Scope-Subset-Verweigerung, Owner-Revoke wirkt sofort, - Audit enthält `parentTokenId`. - -## Offene Fragen - -- Single-Use vs. Multi-Use innerhalb TTL (Shell-Skripte mögen Multi, Audit mag Single)? -- Soll `project_reload`/`backup`/`snapshot` (heute REST-only, `allowAdminActions`) - auch über Ephemeral erreichbar sein oder explizit ausgenommen bleiben? -- Braucht es IP-/User-Agent-Bindung, oder reicht kurze TTL + enger Scope? diff --git a/src/content/docs/server/uebersicht.md b/src/content/docs/server/uebersicht.md deleted file mode 100644 index 72c524f..0000000 --- a/src/content/docs/server/uebersicht.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Tibi-Server Übersicht -description: Was der Tibi-Server kann und wo die Details stehen. ---- - -Der **Tibi-Server** ist ein dynamisches, konfigurationsgetriebenes Headless-CMS in Go: MongoDB plus YAML-definierte REST-APIs, JavaScript-Hooks, SSE-Realtime, LLM-Chat, Upload/Bilder, PDF, Audit, Cron, OpenAPI, Prometheus, Multi-Tenancy. - -## Detailseiten - -Die Detailseiten unter `server/` werden per `npm run sync` aus dem Server-Repo (`tibi-server/docs/`) übernommen und hier nur gerendert — Single Source bleibt das Server-Repo: - -- Architektur, Konfiguration, API-Endpunkte, Collections -- Authentifizierung, Hooks, Realtime, Upload/Bilder -- LLM-Integration, Audit, Jobs, Deployment -- OpenAPI/Metriken, Security, Troubleshooting, Cookbook -- Feld-Permissions, Orgs/Teams, Aktionen, Backup/Restore, Tokens, MCP - -## Konvention für Agenten - -- Neue Server-Funktion → erst Code + `tibi-server/docs/*.md`, dann `npm run sync` hier, dann Build prüfen. -- Generierte Referenz (`reference/`) nie von Hand editieren.