cms/tibi-docs
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial: Tibi-Doku auf Starlight-Basis (Nova-Theme, Docker-Stack, Sync-Skripte)

Browse Source
This commit is contained in:
apairon committed 2026-10-16 11:43:04 +00:00
commit 58502aa3c8
78 files changed
+22860
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/content/docs/server/01-architecture.md
+261
View File
@@ -0,0 +1,261 @@
---
title: 'Architecture'
---
# Architecture
## System Overview
Tibi-Server is a Go-based headless CMS that provides dynamically configured REST APIs backed by MongoDB. Projects are defined via YAML configuration files that specify collections, hooks, jobs, and assets. Each project gets its own MongoDB database (prefixed with the server DB prefix).
```text
┌─────────────────────────────────────────────────────────┐
│ HTTP Clients │
└─────────────┬───────────────────────────────┬───────────┘
│ │
▼ ▼
┌─────────────────────────┐ ┌──────────────────────────┐
│ System API │ │ Collection API │
│ /api/v1/login │ │ /api/v1/_/:project/... │
│ /api/v1/user │ │ │
│ /api/v1/project │ │ Dynamic CRUD per │
│ /api/v1/llm/chat │ │ project config │
│ /api/v1/audit │ │ │
│ /api/v1/version │ │ Admin: │
│ Admin: │ │ /reload │
│ /shutdown │ │ /regenerate-search │
│ /backup, /restore │ │ /backup, /restore │
└─────────┬───────────────┘ └─────────┬────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────┐
│ Gin Router / Middleware │
│ Sentry → Gzip → Prometheus → RouterContext → CORS │
└─────────────────────────┬───────────────────────────────┘
│
┌───────────┴───────────┐
▼ ▼
┌──────────────────────┐ ┌─────────────────────────────┐
│ dbapi (System) │ │ Controller (Collections) │
│ Generic CRUD for │ │ CollectionGet/Post/Put/ │
│ User, Project │ │ Delete + BulkUpdate/ │
│ │ │ BulkDelete with Hooks │
└──────────┬───────────┘ └──────────┬──────────────────┘
│ │
│ ┌────────┴────────┐
│ ▼ ▼
│ ┌──────────────────┐ ┌────────────┐
│ │ Hook Engine │ │ Validator │
│ │ (goja JS) │ │ │
│ │ 20+ packages │ └────────────┘
│ └────────┬─────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────────────────┐
│ dbcrud Layer │
│ Abstract DB Interface │
└─────────────────────────┬───────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ MongoDB Driver │
│ mongocrud implementation │
└─────────────────────────────────────────────────────────┘
```
## Directory Structure
```text
tibi-server/
├── cmd/server/main.go # Entry point: config loading, DB connection, HTTP server
├── internal/
│ ├── config/server.go # ServerConfig struct, YAML + env variable loading
│ ├── controller/ # HTTP handlers
│ │ ├── setup.go # Router setup, route registration, middleware
│ │ ├── collection_get.go # GET handler with filter, projection, pagination
│ │ ├── collection_count.go # COUNT handler for /_count endpoint
│ │ ├── collection_post.go # POST handler with file upload
│ │ ├── collection_put.go # PUT handler with partial update
│ │ ├── collection_delete.go# DELETE handler with file cleanup
│ │ ├── collection_file.go # File serving with image filter support
│ │ ├── collection.go # bindCollection, handleFileInJSON (file upload logic)
│ │ ├── subst.go # $subst expression resolution for permission filters
│ │ ├── validator.go # Unified validateData (3 modes: Create/Update/Changeset)
│ │ ├── admin_reload.go # Hot-reload project config
│ │ ├── assets_get.go # Static asset serving
│ │ ├── audit_get.go # Audit log query endpoint
│ │ ├── llm_chat.go # LLM chat handler (streaming + non-streaming)
│ │ ├── metric.go # Prometheus middleware
│ │ ├── openapi_project_get.go # OpenAPI spec generation
│ │ ├── collection_bulk_update.go # Bulk update (PUT without ID, filter-based)
│ │ ├── collection_bulk_delete.go # Bulk delete (DELETE without ID, filter-based)
│ │ └── operators.go # Computed update operators ($inc, $mul, $unset)
│ ├── dbapi/ # Generic system-level CRUD framework
│ │ ├── handler.go # Generic GET/POST/PUT/DELETE for registered models
│ │ ├── filter.go # MongoDB filter validation and type conversion
│ │ ├── filter_valid_functions.go # Whitelist of allowed MongoDB operators
│ │ ├── login.go # JWT token creation and validation
│ │ └── register.go # Model registration (User, Project)
│ ├── dbcrud/ # Database abstraction layer
│ │ ├── interface.go # Database, Collection, Query interfaces
│ │ ├── model.go # Base Model (ID, InsertTime, UpdateTime, Ident)
│ │ ├── collection.go # Collection name resolution
│ │ └── validator.go # Struct-tag based validation
│ ├── hook/ # JavaScript hook context packages
│ │ ├── compatibility.go # Legacy function name mapping
│ │ ├── context_db.go # db.find, db.create, db.update, db.delete, db.updateMany, ...
│ │ ├── context_http.go # http.fetch, http.fetchStream
│ │ ├── context_fs.go # fs.readFile, fs.writeFile, fs.stat, ...
│ │ ├── context_smtp.go # smtp.sendMail
│ │ ├── context_response.go # response.write, response.writeSSE, response.flush
│ │ ├── context_channel.go # channel.subscribe, channel.send
│ │ └── ... # 20+ context packages total
│ ├── embedding/ # Embedding provider architecture (ONNX Runtime, tokenizer)
│ ├── image/ # Image processing (resize, effects, WebP)
│ ├── llm/ # LLM provider abstraction, quota management
│ ├── models/ # Domain models and business logic
│ │ ├── api.go # API struct: collections, hooks, jobs, CORS, hub
│ │ ├── project.go # Project model, config loading, setup
│ │ ├── collection.go # Collection model, CollectionConfig, fields, permissions
│ │ ├── permission.go # Permission checking logic
│ │ ├── hook.go # Hook execution (run, exception handling)
│ │ ├── eval_context.go # BuildEvalContext / PopulateJsData — central JS context builder
│ │ ├── field_permissions.go # Eval-based readonly/hidden field checks
│ │ ├── auth.go # AuthData extraction from JWT
│ │ ├── user.go # User model with MD5 password hashing
│ │ └── login.go # Login flow, refresh tokens
│ ├── rankfusion/ # Mode-independent Reciprocal Rank Fusion (N-way weighted RRF)
│ ├── realtime/ # SSE hub with channels, subscriptions, ring buffer
│ ├── suexec/ # Thread-based UID/GID isolation for multi-tenancy
│ ├── utils/ # Helpers: error formatting, search, file paths
│ ├── vectorindex/ # In-memory vector index (dense cosine + sparse inverted)
│ └── version/ # Build version info
├── tibi-types/ # TypeScript type definitions for hook context
├── examples/ # Example project configurations
├── vendor/ # Vendored Go dependencies
├── config.yml # Server configuration
├── Dockerfile # Production Docker image
├── Dockerfile.air # Development Docker image (hot-reload)
├── docker-compose.yml # Full stack: server + mongo + maildev + chisel
└── docker-compose-local.yml # MongoDB only for local development
```
## Request Lifecycle
### Collection Request (e.g., GET /api/v1/_/myproject/articles)
```text
1. HTTP Request
│
2. Gin Middleware Chain
├── Sentry (if configured): trace + error tracking
├── Gzip compression (excludes images, videos, PDFs, /metrics)
├── Prometheus: start timing
├── RouterContext: inject global project API context
└── CORS: resolve config (collection → project → server level)
│
3. Route Handler
├── Create new DB session
├── Resolve project namespace from URL path
├── Resolve collection config from URL path
│
4. Permission Check (MethodIsAllowed)
├── Check token-based permissions (header/query)
├── Check public permissions
├── Check admin role (role=0 → always allowed)
├── Check user permissions (role=1 + project assignment)
└── Check custom permissions (namespace:permName)
│
5. Pre-Filter (getCollectionPreFilter)
├── Apply permission filter (e.g., {published: true})
└── Resolve $subst expressions (auth.id, auth.role, etc.)
│
6. Hook Pipeline (per method)
│ GET: request → read → [db query] → return
│ COUNT: read → [db count] (no return hook)
│ POST: bind → validate → create → [db insert] → return
│ PUT: bind → validate → update → [db update] → return
│ DELETE: delete → [db delete] → return
│
├── Each hook step:
│ ├── Load compiled JS program
│ ├── Build hook context (api, project, collection, request, ...)
│ ├── Inject context packages (db, http, fs, smtp, ...)
│ ├── Execute JS with timeout
│ └── Process return value (data override, results override)
│
├── Hook can throw HookException:
│ ├── {status, json} → JSON response
│ ├── {status, html} → HTML response
│ ├── {status, string} → text response
│ ├── {status, bytes} → binary response
│ └── {status, file} → file download
│
7. Database Operation
├── ReadCollection / CreateDocument / UpdateDocument / DeleteDocument
└── File system operations (upload, delete)
│
8. Audit Log (if enabled)
├── Record action, user, document snapshot, changes
└── Write async to audit collection with TTL index
│
9. Response
├── NaN/Inf values converted to null
├── JSON serialization
└── Prometheus: record duration
```
## Data Model
### Base Model (dbcrud.Model)
Every document in the system inherits from the base model:
```go
type Model struct {
ID interface{} // MongoDB ObjectID
InsertTime *time.Time // Set automatically on create
UpdateTime *time.Time // Set automatically on update
Ident []string // Display identifier (e.g., username, document title)
}
```
### System Models
| Model | Database | Collection | Description |
| -------------- | ---------- | ------------- | ---------------------------------------------- |
| `User` | `{prefix}` | `user` | Admin/user accounts with roles and permissions |
| `Project` | `{prefix}` | `project` | Project definitions with config file paths |
| `AuditLog` | `{prefix}` | `auditlog` | System-wide audit trail |
| `LLMUsageLog` | `{prefix}` | `llmusagelog` | LLM token usage tracking |
### Project Models
| Model | Database | Collection | Description |
| -------------- | ---------------------- | ------------------- | ------------------------------ |
| `Collection` | `{prefix}_{namespace}` | `{collectionName}` | Dynamic, schema-less documents |
Each project gets its own database named `{prefix}_{namespace}`, where `prefix` is the server DB prefix (default: `tibi`) and `namespace` is defined in the project's YAML config.
## Middleware Chain
| Order | Middleware | Source | Purpose |
| ----- | ------------- | --------------------------- | ------------------------------------- |
| 1 | Sentry | `sentrygin.New()` | Error tracking, transaction tracing |
| 2 | Sentry Scope | Custom | Set user IP, start spans |
| 3 | Gzip | `gzip.Gzip()` | Compression (excludes binary formats) |
| 4 | Prometheus | `measureResponseDuration()` | Request timing metrics |
| 5 | RouterContext | Custom | Inject project API context |
| 6 | CORS | `cors.New()` | Cross-origin configuration |
## Key Design Decisions
1. **Schema-less collections**: Collection documents use `map[string]interface{}` (BSON inline) rather than fixed Go structs. Field definitions in YAML config are used for validation and type checking but don't restrict storage.
2. **Per-project databases**: Each project gets its own MongoDB database, providing data isolation between projects.
3. **JavaScript hooks via goja**: Hooks run in a Go-embedded JavaScript engine (goja), not Node.js. This means ECMAScript 5.1 compatibility with some ES6+ features. No npm modules available — all functionality comes from built-in context packages.
4. **Suexec isolation**: On Linux, hook execution and file operations run under the UID/GID of the project's config file owner, providing OS-level multi-tenant isolation.
5. **Session-per-request**: Each HTTP request creates a new MongoDB session that is closed after the request completes.
src/content/docs/server/02-configuration.md
+622
View File
@@ -0,0 +1,622 @@
---
title: 'Configuration'
---
# Configuration
Tibi-Server uses a layered configuration system: a YAML config file with environment variable overrides.
## Server Configuration (`config.yml`)
The server configuration file is loaded from the working directory at startup.
### Complete Reference
```yaml
db:
dial: "mongodb://localhost:27017" # MongoDB connection string
prefix: "tibi" # Database name prefix
logging: "" # Enable DB query logging (any non-empty value)
api:
port: 8080 # HTTP server port
maxBodySize: "100MB" # Max request body size (default: 100MB). Supports B, KB, MB, GB, TB.
# Cascade: Server (hard cap) → Project (default) → Collection (override)
# Project-level: set maxBodySize in project config.yml (can only reduce)
# Collection-level: set maxBodySize in collection config.yml (can override project, but not server)
jwtSecret: "your-jwt-secret" # HMAC-SHA256 secret for JWT signing
secureCookies: true # Set Secure flag on cookies (default: true, set false for local HTTP dev)
accessTokenLifetime: "60m" # JWT access token lifetime (Go duration, default: 60m)
refreshTokenLifetime: "10h" # Refresh token / cookie lifetime (Go duration, default: 10h)
adminTokens: # Server-level admin tokens with granular permissions
- token: "agent-token-xyz" # Token string (sent via X-Admin-Token header or Authorization: Bearer)
label: "ci-deploy" # Optional label for audit logging (identifies the token purpose)
permissions: # List of granted permissions
- project # Shorthand for project.* (all CRUD)
- project.reload # Reload project config
- user # Shorthand for user.* (all CRUD)
- server.shutdown # Server shutdown
- namespace.myproject # All collections in myproject (all actions)
- namespace.*.articles.read # Read articles in any project (wildcard)
- token: "readonly-token"
label: "monitoring" # Shows up in audit logs as tokenLabel
permissions:
- project.read # Only read projects
- user.read # Only read users
- namespace.*.*.read # Readonly on all collections in all projects
mail:
host: "smtp.example.com:587" # SMTP server host:port
username: "user@example.com" # SMTP username
password: "password" # SMTP password
security:
allowAbsolutePaths: false # Allow absolute file paths in hooks/config
allowUpperPaths: false # Allow path traversal above config directory
sentry:
dsn: "" # Sentry DSN for error tracking
environment: "production" # Sentry environment tag
serverName: "tibi-1" # Sentry server name
enableTracing: true # Enable performance tracing
debug: false # Sentry debug mode
cors:
allowOrigins: # Allowed origins (server-level default)
- "https://example.com"
allowCredentials: false # Allow credentials (cookies, auth headers)
allowHeaders: # Allowed request headers
- "Content-Type"
- "X-Auth-Token"
allowMethods: # Allowed HTTP methods
- "GET"
- "POST"
- "PUT"
- "DELETE"
exposeHeaders: [] # Headers exposed to the browser
maxAge: 0 # Preflight cache duration (seconds)
response:
error:
message: "" # Fixed error message for clients (hides details)
stack: false # Include stack traces in error responses
llm:
providers:
- name: "openai" # Provider identifier
type: "openai" # "openai" or "openai-compatible"
baseURL: "https://api.openai.com/v1" # API base URL (auto-set for "openai" type)
apiKey: "sk-..." # API key (prefer env variable)
defaultModel: "gpt-4" # Default model
models: # Available models
- "gpt-4"
- "gpt-3.5-turbo"
maxTokensPerRequest: 4096 # Max tokens per request (optional)
audit:
enabled: false # Enable audit logging
defaultTTL: "720h" # Default audit log retention (30 days)
defaultLimit: 50 # Default query limit for audit GET (fallback: 50)
maxLimit: 10000 # Maximum query limit for audit GET (fallback: 10000)
ratelimit:
enabled: true # Enable login rate limiting (default: true)
loginInitialDelay: "1s" # Initial delay after first failed login attempt
loginMaxDelay: "60s" # Maximum delay cap (exponential backoff)
loginResetAfter: "15m" # Reset attempt counter after inactivity period
embedding:
providers:
- name: "bge-m3" # Provider identifier (referenced in collection search config)
type: "native" # "native" (ONNX), "openai", "openai-compatible", "ollama"
modelPath: "/models/bge-m3" # Path to ONNX model directory (type: native)
dimensions: 1024 # Embedding dimensions (required for native)
- name: "e5-small"
type: "native"
modelPath: "/models/multilingual-e5-small"
dimensions: 384
- name: "openai-embed" # Example: external OpenAI provider
type: "openai"
model: "text-embedding-3-small" # Model name for API calls
apiKey: "sk-..." # API key (prefer env variable)
baseURL: "https://api.openai.com/v1" # API base URL
dimensions: 1536
hook:
cache:
maxEntries: 10000 # Max cache entries per project before LRU eviction (default: 10000)
defaultTTL: "0s" # Default TTL for cache.set() when ttl=0 (default: no expiry)
ratelimit:
backoffInitialDelay: "1s" # Backoff initial delay after first failure (default: 1s)
backoffMaxDelay: "60s" # Backoff maximum delay cap (default: 60s)
backoffResetAfter: "15m" # Backoff reset after inactivity (default: 15m)
windowSize: "60s" # Sliding window duration (default: 60s)
windowMaxHits: 60 # Max hits per sliding window (default: 60)
```
### Default Values (no config file)
When no config file is found and `defaultFallback` is true:
| Setting | Default |
| ------------------------ | --------- |
| `api.port` | `8080` |
| `db.prefix` | `"tibi"` |
| `api.maxBodySize` | `"100MB"` |
| `response.error.stack` | `true` |
## Environment Variable Overrides
Environment variables take precedence over config file values.
| Environment Variable | Config Path | Type | Notes |
| ----------------------------- | ----------------------------- | ------ | ----------------------------------- |
| `DB_DIAL` | `db.dial` | string | MongoDB connection string |
| `MGO_DIAL` | `db.dial` | string | Legacy alias for `DB_DIAL` |
| `DB_PREFIX` | `db.prefix` | string | Database name prefix |
| `DB_LOGGING` | `db.logging` | string | Any non-empty value enables logging |
| `API_UPLOAD_MAX_BODY_SIZE` | `api.upload.maxBodySize` | string | e.g. `"100MB"` (B, KB, MB, GB, TB) |
| `API_PORT` | `api.port` | int | HTTP port (panics if ≤ 0) |
| `PORT` | `api.port` | int | Fallback for `API_PORT` |
| `API_JWTSECRET` | `api.jwtSecret` | string | JWT signing secret |
| `API_SECURE_COOKIES` | `api.secureCookies` | bool | Secure flag on cookies (default: true) |
| `API_ACCESS_TOKEN_LIFETIME` | `api.accessTokenLifetime` | string | Go duration, e.g. `"60m"` (default) |
| `API_REFRESH_TOKEN_LIFETIME` | `api.refreshTokenLifetime` | string | Go duration, e.g. `"10h"` (default) |
| `MAIL_HOST` | `mail.host` | string | SMTP host:port |
| `SECURITY_ALLOWABSOLUTEPATHS` | `security.allowAbsolutePaths` | bool | `"true"` or `"false"` |
| `SECURITY_ALLOWUPPERPATHS` | `security.allowUpperPaths` | bool | `"true"` or `"false"` |
| `RESPONSE_ERROR_STACK` | `response.error.stack` | bool | Show stack traces |
| `RESPONSE_ERROR_MESSAGE` | `response.error.message` | string | Fixed error message |
| `AUDIT_ENABLED` | `audit.enabled` | bool | Enable audit logging |
| `AUDIT_DEFAULT_TTL` | `audit.defaultTTL` | string | Go duration (e.g., `"720h"`) |
| `AUDIT_DEFAULT_LIMIT` | `audit.defaultLimit` | int | Audit GET default page size |
| `AUDIT_MAX_LIMIT` | `audit.maxLimit` | int | Audit GET max page size |
| `RATELIMIT_ENABLED` | `ratelimit.enabled` | bool | Enable login rate limiting |
| `RATELIMIT_LOGIN_INITIAL_DELAY` | `ratelimit.loginInitialDelay` | string | Go duration (e.g., `"1s"`) |
| `RATELIMIT_LOGIN_MAX_DELAY` | `ratelimit.loginMaxDelay` | string | Go duration (e.g., `"60s"`) |
| `RATELIMIT_LOGIN_RESET_AFTER` | `ratelimit.loginResetAfter` | string | Go duration (e.g., `"15m"`) |
| `HOOK_CACHE_MAX_ENTRIES` | `hook.cache.maxEntries` | int | Max cache entries per project |
| `HOOK_CACHE_DEFAULT_TTL` | `hook.cache.defaultTTL` | string | Go duration (e.g., `"5m"`) |
| `HOOK_RATELIMIT_BACKOFF_INITIAL_DELAY` | `hook.ratelimit.backoffInitialDelay` | string | Go duration (e.g., `"1s"`) |
| `HOOK_RATELIMIT_BACKOFF_MAX_DELAY` | `hook.ratelimit.backoffMaxDelay` | string | Go duration (e.g., `"60s"`) |
| `HOOK_RATELIMIT_BACKOFF_RESET_AFTER` | `hook.ratelimit.backoffResetAfter` | string | Go duration (e.g., `"15m"`) |
| `HOOK_RATELIMIT_WINDOW_SIZE` | `hook.ratelimit.windowSize` | string | Go duration (e.g., `"60s"`) |
| `HOOK_RATELIMIT_WINDOW_MAX_HITS` | `hook.ratelimit.windowMaxHits` | int | Max hits per sliding window |
| `LLM_<NAME>_APIKEY` | `llm.providers[name].apiKey` | string | Provider API key (uppercase name) |
| `EMBEDDING_<NAME>_APIKEY` | `embedding.providers[name].apiKey` | string | Embedding provider API key (uppercase name) |
| `EMBEDDING_<NAME>_MODELPATH` | `embedding.providers[name].modelPath` | string | ONNX model directory path (uppercase name) |
| `EMBEDDING_<NAME>_DOCUMENTPREFIX` | `embedding.providers[name].documentPrefix` | string | Global prefix applied to document embeddings (uppercase name) |
| `EMBEDDING_<NAME>_QUERYPREFIX` | `embedding.providers[name].queryPrefix` | string | Global prefix applied to query embeddings (uppercase name) |
### LLM API Key Convention
LLM providers can be configured completely via environment variables (without changing `config.yml`) using `LLM_PROVIDERS`:
```bash
# Register two providers
export LLM_PROVIDERS=openai,local-llama
# Configure the "openai" provider
export LLM_OPENAI_APIKEY=sk-...
# Configure the "local-llama" provider
export LLM_LOCAL-LLAMA_TYPE=openai-compatible
export LLM_LOCAL-LLAMA_BASEURL=http://localhost:11434/v1
export LLM_LOCAL-LLAMA_MODELS=llama3,mistral
export LLM_LOCAL-LLAMA_DEFAULTMODEL=llama3
```
All available config fields (`APIKEY`, `TYPE`, `BASEURL`, `DEFAULTMODEL`, `MODELS`) can be overridden via `LLM_<UPPERCASENAME>_<FIELD>`.
### Embedding Provider Environment Convention
Embedding providers are configured in exactly the same way as LLM providers. Rather than `llm:`, you provide an `embedding:` block in your server's `config.yml`.
Example in `config.yml`:
```yaml
embedding:
providers:
- name: bge-m3
type: native
model: bge-m3
dimensions: 1024
# modelPath: /assets/models/bge-m3
# documentPrefix: ""
# queryPrefix: ""
- name: text-embedding-ada-002
type: openai
model: text-embedding-ada-002
dimensions: 1536
# apiKey: sk-...
```
Just like LLMs, you can provision Embedding providers **purely via environment variables** (very useful for Docker setups):
```bash
# Register two providers via comma-separated list
export EMBEDDING_PROVIDERS=openai-embed,bge-m3
# Configure "openai-embed"
export EMBEDDING_OPENAI-EMBED_TYPE=openai
export EMBEDDING_OPENAI-EMBED_MODEL=text-embedding-ada-002
export EMBEDDING_OPENAI-EMBED_DIMENSIONS=1536
export EMBEDDING_OPENAI-EMBED_APIKEY=sk-...
# Configure "bge-m3"
export EMBEDDING_BGE-M3_TYPE=native
export EMBEDDING_BGE-M3_MODELPATH=/models/bge-m3
export EMBEDDING_BGE-M3_DIMENSIONS=1024
export EMBEDDING_BGE-M3_DOCUMENTPREFIX="search_document: "
export EMBEDDING_BGE-M3_QUERYPREFIX="search_query: "
```
Available config fields: `TYPE`, `MODEL`, `MODELPATH`, `DIMENSIONS`, `BASEURL`, `APIKEY`, `DOCUMENTPREFIX`, `QUERYPREFIX`.
**Note:** The collections within your projects will reference these providers by their `name`.
Default type for embedding providers (if omitted in config) is `"native"` (ONNX).
---
## Project Configuration
Each project has a YAML config file referenced in the project's `configFile` field in the database. The config file defines all collections, hooks, jobs, and assets for that project.
### Complete Project Reference
```yaml
# Project namespace (used as database suffix and URL path segment)
# Must match: [a-zA-Z0-9_]+
namespace: myproject
# Arbitrary metadata accessible in hooks via context.project().meta
meta:
title: "My Project"
version: "1.0"
# Collection definitions
collections:
- name: articles # Collection name (URL path segment, MongoDB collection name)
defaultLanguage: "german" # Default language for text indexes
uploadPath: "./media" # File upload directory (relative to config dir)
fields: # Field definitions (see Collections docs)
- name: title
type: string
validator:
required: true
- name: image
type: file
validator:
maxFileSize: "5MB" # Max file size for this field (supports B, KB, MB, GB, TB)
permissions: # Permission sets (see Authentication docs)
public:
methods:
get: true
filter:
published: true
user:
methods:
get: true
post: true
put: true
delete: true
projections: # Named projections (field subsets)
default:
select:
title: 1
image: 1
detailed:
select:
title: 1
image: 1
content: 1
imageFilter: # Named image filter presets
thumbnail:
- width: 200
height: 200
fit: true
quality: 80
hero:
- width: 1200
height: 600
fill: true
anchor: center
hooks: # Lifecycle hooks (see Hooks docs)
get:
request:
type: javascript
file: "hooks/get_request.js"
timeout: 5000
post:
create:
type: javascript
file: "hooks/post_create.js"
indexes: # Custom MongoDB indexes
- name: title_text
key:
- "$text:title"
defaultLanguage: german
- name: date_sort
key:
- "-publishDate"
background: true
strictFields: true # Reject unknown fields not in schema (default: inherit from project)
cors: # Collection-level CORS override
merge: true
allowOrigins:
- "https://special.example.com"
audit: # Collection-level audit config
enabled: true
actions:
- create
- update
- delete
queryLimits: # Query result limits for GET list endpoint
defaultLimit: 100 # Default limit when no ?limit= param (fallback: 10000)
maxLimit: 500 # Maximum allowed limit (fallback: 10000)
audit: # Project-level audit (opt-in, default false)
enabled: true # required in addition to server audit.enabled
# Cron jobs
jobs:
- cron: "0 */5 * * *" # Every 5 minutes
type: javascript
file: "jobs/cleanup.js"
timeout: 30000 # Milliseconds (30 s)
meta:
description: "Cleanup old drafts"
# Static asset directories
assets:
- name: static
path: "./public"
- name: uploads
path: "./uploads"
# Reject unknown fields for all collections by default (default: false)
strictFields: true
# Project-level CORS configuration
cors:
merge: true # Merge with server CORS (true) or replace (false)
allowOrigins:
- "https://myapp.example.com"
allowCredentials: true
allowHeaders:
- "X-Custom-Header"
# Admin configuration
admin:
tokens:
- token: "deploy-token-abc123"
allowReload: true
# Hook settings (project-level overrides)
hook:
cache:
maxEntries: 5000 # Override server default for this project
defaultTTL: "5m" # Override default TTL for this project
ratelimit:
backoffInitialDelay: "2s" # Override backoff initial delay
backoffMaxDelay: "120s" # Override backoff max delay
backoffResetAfter: "30m" # Override backoff reset period
windowSize: "30s" # Override sliding window duration
windowMaxHits: 30 # Override max hits per window
# Legacy compatibility settings
legacy:
httpCodes: true # true (default): use original status codes for backward compatibility
# false: use RFC-compliant HTTP status codes (201, 204, 401, 409, 422)
```
### Environment Variable Substitution in Project Config
Project config files support a companion `.env` file. If `config.yml.env` exists alongside `config.yml`, its variables are substituted into the YAML before parsing:
```bash
# config.yml.env
API_KEY=secret-key-123
WEBHOOK_URL=https://example.com/webhook
```
```yaml
# config.yml - use ${VAR} syntax
meta:
apiKey: "${API_KEY}"
webhookUrl: "${WEBHOOK_URL}"
```
### YAML Include Support
Project configs support including other YAML files for modular collection definitions:
```yaml
# config.yml
namespace: myproject
collections: !include collections/*.yml
```
This allows splitting collection configs into separate files under a `collections/` directory.
---
## Legacy HTTP Status Codes
By default, Tibi-Server uses legacy HTTP status codes for backward compatibility. To opt into RFC-compliant status codes, set `legacy.httpCodes: false` in the project config:
```yaml
legacy:
httpCodes: false
```
### Status Code Mapping
| Scenario | Legacy (`true`, default) | Modern (`false`) |
|---|---|---|
| POST/create success | 200 OK | **201** Created |
| Single DELETE success | 200 OK + body | **204** No Content |
| Authentication missing | 403 Forbidden | **401** Unauthorized |
| Validation error | 400 Bad Request | **422** Unprocessable Entity |
| Duplicate key conflict | 500 Internal Server Error | **409** Conflict |
| Field permission denied (readonly/hidden) | 400 Bad Request | **403** Forbidden |
**Notes:**
- Bulk DELETE always returns 200 with `{"deleted": N}` regardless of the setting.
- Bulk POST returns 201 in modern mode (with `{"created": N, "ids": [...]}`).
- The default is `true` — existing projects are unaffected unless explicitly opted in.
- The setting is per-project, allowing gradual migration.
---
## CORS Configuration Hierarchy
CORS is configured at three levels, with each level optionally merging with or replacing the parent:
```text
Server CORS (config.yml → cors)
│
├── merge: false → Replace entirely
│
▼
Project CORS (project config → cors)
│
├── merge: true → Append to server origins/headers/methods
├── merge: false → Replace entirely
│
▼
Collection CORS (collection config → cors)
│
└── merge: true → Append to project+server origins/headers/methods
```
### Resolution Order (per request)
1. Check if there's a collection-specific CORS config → `CORSMap["/collectionName"]`
2. Fall back to project-level CORS → `CORSMap["/"]`
3. Fall back to server-level CORS → `CORSMap[""]`
### Merge Behavior
When `merge: true`:
- `allowOrigins` are **appended** to the parent config
- `allowHeaders` are **appended** to the parent config
- `allowMethods` are **appended** to the parent config
- `exposeHeaders` are **replaced** (not appended — see [ISSUES.md](ISSUES.md))
- `allowCredentials` is **replaced**
- `maxAge` is **set to 0** (always)
- Duplicates are automatically removed
### CORSConfig Struct
```go
type CORSConfig struct {
Merge bool `yaml:"merge"`
AllowOrigins []string `yaml:"allowOrigins"`
AllowCredentials bool `yaml:"allowCredentials"`
AllowHeaders []string `yaml:"allowHeaders"`
AllowMethods []string `yaml:"allowMethods"`
ExposeHeaders []string `yaml:"exposeHeaders"`
MaxAge int `yaml:"maxAge"`
}
```
---
## Collection Configuration Reference
### CollectionConfig Struct
```go
type CollectionConfig struct {
Name string // Collection/MongoDB collection name
DefaultLanguage *string // Default language for text indexes
UploadPath *string // File upload directory (relative or absolute)
Permissions map[string]PermissionSet // Permission sets (public, user, token:*, custom)
Projections map[string]ProjectionConfig // Named field projections
Meta map[string]interface{} // Arbitrary metadata
Fields []CollectionField // Field definitions
ImageFilter map[string][]image.FilterParams // Named image filter presets
Hooks map[string]map[string]HookConfig // Lifecycle hooks [method][step]
Indexes []CollectionIndex // Custom MongoDB indexes
CORS *CORSConfig // Collection-level CORS
Audit *AuditCollectionConfig // Collection-level audit config
Bulk *BulkConfig // Bulk operation config
QueryLimits *QueryLimitsConfig // Query result limits (defaultLimit, maxLimit)
}
```
### CollectionIndex
```yaml
indexes:
- name: "unique_email" # Index name
key: # Index keys (prefix with - for descending, $text: for text)
- "email"
unique: true # Unique constraint
sparse: true # Skip documents without the indexed field
background: true # Build in background
dropDups: false # Drop duplicate entries (deprecated in MongoDB 3.0+)
defaultLanguage: "english" # Language for text indexes
languageOverride: "lang" # Field that overrides the language
```
### Field-Level Indexes
Fields can define indexes directly via the `index` property:
```yaml
fields:
- name: email
type: string
index:
- single # Regular index on this field
- unique # Unique index
- text # Text search index
- sparse # Sparse index
```
### Index Auto-Management
On project setup/reload:
1. All configured indexes are created via `EnsureIndex`
2. Indexes that exist in MongoDB but are no longer in the config are automatically dropped (`fixIndexes`)
3. The `_id_` index is never dropped
---
## Admin Configuration
### Project-Level Admin Tokens
Each project can define admin tokens for project-specific operations:
```yaml
admin:
tokens:
- token: "deploy-token-abc123"
allowReload: true
allowRegenerateSearch: true
```
Available permissions:
- `allowReload` – allows reloading the project configuration
- `allowRegenerateSearch` – allows regenerating search indexes (`_search` data)
Usage:
```bash
curl -X POST http://localhost:8080/api/v1/_/myproject/_/admin/reload \
-H "Authorization: Bearer deploy-token-abc123"
```
src/content/docs/server/03-api-endpoints.md
+2410
View File
File diff suppressed because it is too large. Load diff
src/content/docs/server/04-collections.md
+1749
View File
File diff suppressed because it is too large. Load diff
src/content/docs/server/05-authentication.md
+606
View File
@@ -0,0 +1,606 @@
---
title: 'Authentication & Authorization'
---
# Authentication & Authorization
## Overview
Tibi-Server uses JWT-based authentication with a role and permission system. Access control is configured per collection via permission sets that support public access, role-based access, token-based access, and custom permissions.
## Login Flow
### Username/Password Login
```
POST /api/v1/login
Content-Type: application/json
{"username": "admin", "password": "admin"}
```
1. Password is hashed with MD5 and compared against the stored hash
2. If matched, a JWT token and refresh token are generated
3. JWT token is returned in the response body
4. Refresh token is set as an HttpOnly cookie
### Refresh Token Login
```
POST /api/v1/login
Cookie: tibi-refresh-token=eyJ...
```
When no username/password is provided but a refresh token cookie exists:
1. The refresh token JWT is validated (HMAC-SHA256, expiration required)
2. User ID is extracted from the token claims
3. User is looked up by ID
4. New JWT token is issued
### Logout
```
POST /api/v1/login?logout=1
```
Clears the refresh token cookie (sets expiry to -1). Any non-empty `logout` query parameter value triggers logout.
### Impersonation (Admin/OrgAdmin → User)
```
POST /api/v1/login/impersonate
X-Auth-Token: <admin-jwt>
Content-Type: application/json
{"userId": "<target-user-id>"}
```
Allows a SystemAdmin (role=0) or OrgAdmin to obtain a JWT token acting as another user.
This is useful for debugging permission issues ("why can User X not see document Y?").
**Security rules:**
| Role | Can impersonate | Reason |
|---|---|---|
| SystemAdmin (0) | Any user | Admin already has full access |
| OrgAdmin | Users in same org(s) | OrgAdmin can already grant themselves the same team permissions |
| User (1) | Nobody | Would break the permission model |
**Response (200):**
```json
{
"token": "<jwt-acting-as-target-user>",
"user": { "id": "...", "username": "targetuser", "role": 1, ... },
"impersonatedBy": {
"id": "<original-admin-id>",
"username": "admin",
"role": 0
}
}
```
**Important:** The JWT token contains an `impersonatedBy` field in the `object` claim,
which is used by the audit system to record who actually performed each action.
Every audit entry during an impersonated session will include:
- `userId`/`username`/`userRole` → the impersonated (target) user
- `impersonatedBy` → the original admin who initiated the impersonation
- `authMethod` → `"impersonation"`
**Logout from impersonation:** Same as normal logout (`POST /api/v1/login?logout=1`).
The refresh token cookie is cleared, ending the impersonated session. The admin
must re-authenticate with their own credentials.
**Chained impersonation:** If an admin impersonates an org-admin, and that
org-admin then impersonates a user, the `impersonatedBy` field always points to
the **original admin** who started the chain — not the intermediate impersonator.
**Errors:**
| Status | Description |
|---|---|
| 400 | `userId is required` or invalid userId format |
| 403 | `impersonation not allowed: insufficient role` |
| 403 | `impersonation not allowed: not admin of target user's org` |
| 404 | target user not found |
## JWT Token Structure
### Access Token
| Field | Value | Description |
|-------|-------|-------------|
| Algorithm | HS256 | HMAC-SHA256 |
| `object` | User data | `{id, username, role, permissions, orgs, teams, primaryOrgId, meta, ...}` |
| `type` | `"User"` | Token type identifier |
| `exp` | now + 60min | Expiration timestamp |
The `password` field is explicitly excluded from the token (via custom JSON marshaling).
The `object` includes org and team membership if the user has any:
```json
{
"object": {
"id": "60a1b2c3d4...",
"username": "alice",
"role": 1,
"permissions": [],
"orgs": [{"orgId": "...", "role": "admin"}],
"teams": [{"teamId": "...", "role": "member"}],
"primaryOrgId": "..."
},
"type": "User",
"exp": 1700000000
}
```
See [Orgs & Teams](18-orgs-teams.md) for details on org/team-based access control.
**Auth Header:**
```
X-Auth-Token: eyJhbGciOiJIUzI1NiIs...
```
### Refresh Token
| Field | Value | Description |
|-------|-------|-------------|
| Algorithm | HS256 | HMAC-SHA256 |
| `object` | `{id: "userId"}` | Minimal user reference |
| `type` | `"RefreshTokenData"` | Token type identifier |
| `exp` | now + 10h | Expiration timestamp |
**Cookie:**
| Property | Value |
|----------|-------|
| Name | `tibi-refresh-token` |
| MaxAge | 36000 (10 hours) |
| Path | `/` |
| HttpOnly | `true` |
| SameSite | `Strict` |
| Secure | `false` |
### Token Validation
Token validation includes caching — subsequent calls within the same request reuse the validated result:
```go
// Cached in gin.Context:
// "validate-auth-token-object" → user data map
// "validate-auth-token-type" → token type string
// "validate-auth-token-err" → validation error
```
---
## Admin Tokens (Server-Level)
Admin tokens provide token-based access for agents, CI/CD pipelines, and other automated systems without requiring the JWT login flow.
### Configuration
```yaml
api:
adminTokens:
- token: "agent-token-xyz"
label: "ci-deploy" # Optional: human-readable label for audit logs
permissions:
- project # Shorthand: all project CRUD operations
- project.reload # Reload project config
- user # Shorthand: all user CRUD operations
- server.shutdown
- namespace.myproject # All collections in myproject
- token: "readonly-token"
label: "monitoring" # Appears as tokenLabel in audit entries
permissions:
- project.read
- user.read
- namespace.*.*.read # Read-only across all namespaces and collections
- token: "importer-token"
label: "data-import"
permissions:
- namespace.myproject.articles.create # Only create in myproject/articles
- namespace.myproject.articles.update # Only update in myproject/articles
```
### Available Permissions
| Permission | Description |
|-----------|-------------|
| `project` | Shorthand for all project operations (`project.*`) |
| `project.read` | GET projects |
| `project.create` | POST (create) projects |
| `project.update` | PUT (update) projects |
| `project.delete` | DELETE projects |
| `project.reload` | Reload project configuration |
| `user` | Shorthand for all user operations (`user.*`) |
| `user.read` | GET users |
| `user.create` | POST (create) users |
| `user.update` | PUT (update) users |
| `user.delete` | DELETE users |
| `server.shutdown` | Shutdown the server |
| `namespace` | Shorthand: all collections in all projects |
| `namespace.{project}` | Shorthand: all collections in a project |
| `namespace.{project}.{collection}` | Shorthand: all actions on a collection |
| `namespace.{project}.{collection}.read` | GET documents |
| `namespace.{project}.{collection}.create` | POST (create) documents |
| `namespace.{project}.{collection}.update` | PUT (update) documents |
| `namespace.{project}.{collection}.delete` | DELETE documents |
### Wildcards
Use `*` as a wildcard for any single segment in a permission string:
| Permission | Description |
|-----------|-------------|
| `namespace.*` | All collections in any project |
| `namespace.*.articles.read` | Read articles across all projects |
| `namespace.myproject.*.read` | Read all collections in myproject |
| `namespace.*.*.read` | Read-only across everything |
Shorthand works at any level — a permission with fewer segments implies all descendants:
- `namespace` = `namespace.*.*.*` (everything)
- `namespace.myproject` = `namespace.myproject.*.*` (all collections, all actions)
- `namespace.myproject.articles` = `namespace.myproject.articles.*` (all actions)
### Usage
Send the token via `X-Admin-Token` header or `Authorization: Bearer` header:
```
# Via X-Admin-Token header — project management
POST /api/v1/project
X-Admin-Token: agent-token-xyz
Content-Type: application/json
{"name": "my-project", "description": "A new project", "configFile": "/path/to/config.yml"}
```
```
# Via Authorization: Bearer header — collection access
GET /api/v1/_/myproject/articles
Authorization: Bearer readonly-token
```
```
# Create document in a collection
POST /api/v1/_/myproject/articles
X-Admin-Token: importer-token
Content-Type: application/json
{"title": "New Article", "body": "Content..."}
```
### Token Priority
For project/user endpoints:
1. **Admin Token** (`X-Admin-Token` / `Authorization: Bearer`): Checked first. If present and valid, access is granted immediately. If present but invalid, access is denied (no fallthrough).
2. **JWT Auth** (`X-Auth-Token`): Checked as fallback if no admin token is provided.
For collection endpoints (`/_/:project/:collection`):
1. **Admin Token**: If present, checked for `namespace.{project}.{collection}.{action}` permission. If present but lacking permission, access is denied (no fallthrough to JWT or public permissions).
2. **Collection Token Permissions**: Header/query-based tokens defined in collection config.
3. **Public / User / JWT**: Normal permission chain if no admin token is present.
### Project Admin Endpoints
For project admin endpoints (`reload`, `regenerate-search`, `backup`, `restore`), three auth methods are supported (checked in order):
1. Server-level admin tokens with the corresponding permission (e.g. `project.reload`, `project.regenerate-search`, `project.backup`)
2. Project-level tokens configured in the project's `config.yml` (`admin.tokens[].allowReload`, `admin.tokens[].allowRegenerateSearch`, `admin.tokens[].allowBackup`)
3. JWT login with admin role (`X-Auth-Token` header)
### Audit Trail
All admin token actions are logged in the audit log with additional context:
| Field | Description | Example |
|-------|-------------|---------|
| `authMethod` | Authentication method used | `admin-token`, `jwt` |
| `tokenLabel` | Label from token config (if set) | `ci-deploy` |
| `tokenPrefix` | First 8 characters of the token | `agent-to` |
| `ip` | Client IP address | `192.168.1.100` |
The full token is **never** logged for security reasons. Use descriptive `label` values to identify tokens in audit logs.
---
## Roles
| Role | Value | Description |
|------|-------|-------------|
| Admin | `0` | Full access to all system and collection endpoints |
| User | `1` | Access limited to assigned projects and permitted methods |
### Admin Role Behavior
- Can access all projects regardless of assignment
- Can manage all users (create, update, delete)
- Can manage all projects
- Bypasses all collection permission checks
- No projection restrictions
- No pre-filter application
- Only admins can change `role`, `permissions`, and `llmBudgets` on users
### User Role Behavior
- Can only see projects belonging to their orgs (via `project.orgId` ∈ user's org IDs)
- Cannot create/update/delete projects (OrgAdmins can modify `teams[]`)
- Can only GET and PUT their own user profile (OrgAdmins have broader scope)
- Cannot change `username`, `role`, `permissions`, or `llmBudgets`
- Subject to collection permission checks (via team permissions)
- See [Orgs & Teams](18-orgs-teams.md) for the full role hierarchy
---
## Permission System
Each collection defines a `permissions` map with named permission sets:
```yaml
permissions:
public:
methods:
get: true
filter:
published: true
user:
methods:
get: true
post: true
put:
allow: true
bulk: true # allow bulk PUT
delete:
allow: true
bulk: true # allow bulk DELETE
"token:apikey":
methods:
get: true
post: true
editor:
methods:
get: true
put: true # shorthand for {allow: true}
```
### PermissionSet Structure
```go
type PermissionSet struct {
Methods map[string]MethodPermission // Allowed methods (get, post, put, delete)
Filter dbcrud.M // Auto-applied filter for queries
ValidProjections []string // Allowed projection names
CompiledFilterSubs []*CompiledSubst // Pre-compiled $subst expressions
}
// MethodPermission represents a method permission.
// Can be unmarshalled from a boolean (true/false) or an object {allow: bool, bulk: bool}.
type MethodPermission struct {
Allow bool // Allow the method
Bulk bool // Allow bulk operations (POST array, PUT/DELETE without ID)
}
```
### Permission Check Priority
When a request comes in, permissions are checked in this order:
```
1. Token-based permissions (header/query parameter match)
│
├── Token name found, value matches → Use token's permission set
├── Token name found, value WRONG → ACCESS DENIED (no fallback!)
└── No token name found → Continue
│
2. Public permissions
│
├── Method allowed → ACCESS GRANTED (no auth required)
└── Method not allowed → Continue
│
3. Auth token validation (X-Auth-Token header)
│
├── No token / invalid → ACCESS DENIED
└── Valid token → Continue
│
4. Admin role check (role = 0)
│
├── Is admin → ACCESS GRANTED
└── Not admin → Continue
│
5. Team "user" permissions (role = 1, team-assigned to project)
│
├── User is member of a team assigned to this project
│ AND collection has "user" permission set → Use "user" set
└── Otherwise → Continue
│
6. Custom user permissions
│
├── Auth permissions contain "namespace:permName" → Use that permission set
└── No match → Continue
│
7. Custom team permissions
│
├── User's teams (assigned to project) have permissions matching a collection key
│ → Use that team permission set (union of all matching teams)
└── No match → ACCESS DENIED
```
See [Orgs & Teams — Permission System](18-orgs-teams.md#permission-system) for details on team-based permission resolution.
### Token-Based Permissions
Token permissions are matched against HTTP headers or query parameters:
```yaml
permissions:
"token:secretkey123": # Matches ?token=secretkey123 OR header token: secretkey123
methods:
get: true
"x-api-key:my-api-key": # Matches ?x-api-key=my-api-key OR header x-api-key: my-api-key
methods:
get: true
post: true
```
**Important:** The permission key format is `name:value`. The `name` part is checked against both query parameters and headers. If the `name` is found but the `value` doesn't match, access is **denied immediately** with no fallback to public/user permissions.
### Custom Permissions
Users can have custom permission strings in their `permissions` array:
```json
{
"username": "editor1",
"role": 1,
"permissions": ["myblog:editor", "myblog:moderator"]
}
```
These are matched against collection permission keys using the format `{namespace}:{permissionKey}`:
```yaml
# In project config (namespace: myblog)
permissions:
editor: # Matched by "myblog:editor"
methods:
get: true
put: true
moderator: # Matched by "myblog:moderator"
methods:
get: true
delete: true
```
---
## Pre-Filters
Permission sets can define filters that are automatically applied to all queries:
```yaml
permissions:
public:
methods:
get: true
filter:
published: true
deletedAt: null
user:
methods:
get: true
post: true
put: true
filter:
author:
$subst:
expr: "auth.id" # Only see own documents
```
### How Pre-Filters Work
1. The permission set's filter is deep-copied (to avoid mutation)
2. `$subst` expressions are resolved with the current request context
3. The filter is combined with any user-provided filter query via `$and`
4. Admin role (0) never has pre-filters applied
### Filter Combination
```
Final filter = $and: [
{ _id: <from URL if single document> },
{ <pre-filter from permissions> },
{ <user-provided filter query> }
]
```
---
## Projection Restrictions
Permission sets can restrict which projections are available:
```yaml
permissions:
public:
methods:
get: true
validProjections:
- summary # Public can only use "summary" projection
user:
methods:
get: true
validProjections:
- summary
- detailed # Users can also use "detailed"
```
**Behavior:**
- If `validProjections` is empty/not set → no restrictions (any projection or none)
- If `validProjections` is set and no `projection` parameter → first entry is used as default
- If `validProjections` is set and invalid projection requested → `400 Bad Request`
- Admin role (0) is never restricted
---
## AuthData Model
Extracted from the JWT token on each authenticated request:
```go
type AuthData struct {
ID string // User ObjectID (hex)
Username string // Username
Role int // 0=Admin, 1=User
Permissions []string // Custom permission strings
}
```
---
## User Model
```go
type User struct {
Model // ID, InsertTime, UpdateTime
Username *string // Required, alphanumeric, unique
Password *string // Required, min 3 chars (stored as MD5 hash)
Role *int // 0=Admin, 1=User (validated min=0, max=2)
Permissions []string // Custom permission strings
Meta map[string]interface{} // Arbitrary user metadata
LLMBudgets []LLMBudget // Per-provider LLM token budgets
}
```
### User Search
The `_search` parameter on `/api/v1/user` searches across:
- `username`
- `meta.company`
- `meta.prename`
- `meta.surname`
- `meta.email`
- `meta.phone`
---
## Server-Level Admin Authentication
Server-level admin endpoints (e.g. shutdown) are protected by admin tokens with the appropriate permission:
```bash
curl -X POST http://localhost:8080/api/v1/admin/shutdown \
-H "X-Admin-Token: your-admin-token"
```
Admin tokens are configured via `api.adminTokens` in `config.yml`. See [Configuration](02-configuration.md) for details.
## Personal Access Tokens (PAT)
Users can create and manage their own API tokens. A PAT uses the same
`X-Auth-Token` header as the JWT and is always checked live against the owner's
current rights. Tokens are a core feature and always enabled.
See [22-personal-access-tokens.md](22-personal-access-tokens.md) for format,
scope grammar, endpoints, examples and security properties.
src/content/docs/server/06-hooks.md
+1721
View File
File diff suppressed because it is too large. Load diff
src/content/docs/server/07-realtime.md
+190
View File
@@ -0,0 +1,190 @@
---
title: 'Realtime Channels (SSE Pub/Sub)'
---
# Realtime Channels (SSE Pub/Sub)
## Overview
Tibi-Server includes an in-memory Pub/Sub hub for real-time communication via **Server-Sent Events (SSE)**. Each project has its own isolated `Hub` instance. Channels are created on-demand when the first subscriber connects, and automatically garbage-collected when the last subscriber leaves.
## Architecture
```
Client A ─── SSE ──→ [Subscription] ──→ [Channel "chat"] ←── [Subscription] ←── SSE ──── Client B
│
[Ring Buffer]
│
Client C ─── SSE ──→ [Subscription] ──→ [Channel "chat"]
│
Hook/Job ────────→ hub.Send("chat", data)
```
### Components
| Component | Description |
|-----------|-------------|
| **Hub** | Per-project container for all channels. Thread-safe via `sync.RWMutex`. |
| **Channel** | Named pub/sub channel with a fixed-size ring buffer. Created on first subscribe. |
| **Subscription** | Per-client connection to a channel. Each gets independent read position. |
| **Message** | Wrapper for data with `CreatedAt` timestamp. |
## Data Structures
### Message
```go
type Message struct {
Data interface{}
CreatedAt time.Time
}
```
### SubscribeOptions
```go
type SubscribeOptions struct {
BufferSize int // Ring buffer capacity (default: 256, set by first subscriber)
OnFull OnFullStrategy // What to do when subscriber is slow
MessageTTL time.Duration // Skip messages older than this (0 = no TTL)
LastN int // Replay last N messages on connect (0 = none)
MaxAge time.Duration // Replay messages newer than this (0 = none)
}
```
### OnFullStrategy
| Strategy | Value | Behavior |
|----------|-------|----------|
| `DropOldest` | `0` (default) | Slow subscriber's read pointer jumps forward (may miss messages) |
| `DropNewest` | `1` | New message is silently discarded for this subscriber |
## Ring Buffer
The channel uses a fixed-size ring buffer for message storage:
```
Buffer size: 8
Write pointer: writeSeq (monotonically increasing)
Position = writeSeq % size
[0] [1] [2] [3] [4] [5] [6] [7]
↑ ↑
oldest writeSeq
```
- **First subscriber** determines the buffer size (subsequent subscribers use the existing size)
- Messages are deep-copied to each subscriber (independent mutation)
- `oldest = max(0, writeSeq - size)` — positions older than this have been overwritten
## Replay Logic
When a new subscriber connects with `LastN` or `MaxAge`:
1. `readSeq` starts at `writeSeq` (only new messages by default)
2. If `LastN > 0`: candidate = `writeSeq - LastN`, clamped to `oldest`
3. If `MaxAge > 0`: backward scan from `writeSeq` until a message older than `MaxAge` is found
4. The lower value of both is used, but never below `oldest`
This means subscribers immediately receive matching historical messages.
## Message TTL
If `MessageTTL` is set on a subscription, messages older than the TTL are silently skipped during `tryRead()`. This prevents slow subscribers from receiving stale data after reconnection.
## Usage from Hooks
### Subscribe (Blocking SSE Endpoint)
```js
// hooks/events/get.before
context.response.header("Content-Type", "text/event-stream");
context.response.header("Cache-Control", "no-cache");
context.response.header("Connection", "keep-alive");
context.response.writeHeader(200);
var sub = channel.subscribe("notifications", {
bufferSize: 100, // Ring buffer size
onFull: "drop-newest", // "drop-oldest" (default) or "drop-newest"
messageTTL: 60000, // 60 seconds
lastN: 10, // Replay last 10 messages
maxAge: 30000 // Only replay messages from last 30 seconds
});
var msg;
while ((msg = sub.receive()) !== null) {
context.response.writeSSE("message", context.json.stringify(msg));
}
sub.close();
throw { status: 200 };
```
### Send (From Any Hook or Job)
```js
// hooks/posts/post.after
context.channel.send("notifications", {
type: "new_post",
id: context.data.id,
title: context.data.title,
timestamp: new Date().toISOString()
});
```
## Receive Behavior
| Return Value | Meaning |
|-------------|---------|
| `(data, nil)` | Message received (deep-copied) |
| `null` | Client disconnected (request context cancelled) |
| Error | Subscription, channel, or hub was closed |
`receive()` blocks until:
- A new message arrives
- The client disconnects
- The subscription/channel/hub is closed
## Automatic Cleanup
- `channel.subscribe()` calls `DisableTimeout()` — the hook timeout is disabled
- A cleanup function is registered via `RegisterCleanup()` — the subscription is closed when the hook ends
- When the last subscriber leaves a channel, the channel is removed from the hub
## Client-Side SSE
```javascript
const eventSource = new EventSource('/api/v1/myproject/events');
eventSource.addEventListener('message', function(e) {
const data = JSON.parse(e.data);
console.log('Received:', data);
});
eventSource.onerror = function(e) {
console.log('Connection lost, reconnecting...');
};
```
## SSE Wire Format
Messages sent via `response.writeSSE()` follow the standard SSE format:
```
event: message
data: {"type":"new_post","id":"abc123"}
event: message
data: {"type":"update","id":"def456"}
```
Each message is terminated by `\n\n` and flushed immediately.
## Limitations
- **In-memory only** — messages are lost on server restart
- **Single-instance** — no cross-server channel synchronization
- **No persistence** — no message durability guarantees
- **Buffer overflow** — slow subscribers may lose messages (ring buffer wrap-around)
- **No authentication on channels** — access is controlled by the collection's permission system that the hook runs in
src/content/docs/server/08-file-upload-images.md
+292
View File
@@ -0,0 +1,292 @@
---
title: 'File Uploads & Image Processing'
---
# File Uploads & Image Processing
## File Upload Mechanism
Files are uploaded as **Base64 data URIs** embedded in JSON document fields of type `file` or `file[]`.
### Upload Format
```json
{
"avatar": "data:image/jpeg;base64,/9j/4AAQSkZJRg...",
"documents": [
"data:application/pdf;base64,JVBERi0xLjQ...",
"data:text/plain;base64,SGVsbG8gV29ybGQ="
]
}
```
### How It Works
1. **POST/PUT request** contains a `file` field with a Base64 data URI
2. The server detects the `data:` prefix in `handleFileInJSON()`
3. The Base64 content is decoded
4. A file is written to: `{uploadPath}/{documentId}/{fieldName}/{filename}`
5. The data URI in the document is replaced with the relative file path
6. The stored document contains only the path reference, not the binary data
### File Path Structure
```
{uploadPath}/
{documentId}/
{fieldName}/
{originalFilename}
```
- `uploadPath` is configured per collection (default: derived from project config)
- File permissions: `0755`
- Operations run under the project's configured UID (suexec)
### Retrieving Files
```
GET /api/v1/{namespace}/{collection}/{id}/{field}/{filename}
```
Example:
```
GET /api/v1/blog/posts/60a1b2c3d4e5f6/avatar/photo.jpg
```
### File Deletion
When a document is deleted, all associated files in `{uploadPath}/{documentId}/` are removed automatically via `os.RemoveAll`.
### Size Limits
**Request Body Size** (Middleware — checked before any processing):
- Server-level: `api.maxBodySize` (default: 100MB) — absolute hard cap
- Project-level: `maxBodySize` in project config — default for all collections
- Collection-level: `maxBodySize` in collection config — overrides project (can increase, but never above server)
- Returns **413** if exceeded
**Per-Field File Size** (checked during Base64 decode):
```yaml
fields:
- name: avatar
type: file
validator:
maxFileSize: "500KB" # Supports B, KB, MB, GB, TB
```
- Uses `io.LimitReader` for memory-efficient streaming check
- Returns **400** if exceeded (file is deleted from disk)
- Works for both `file` and `file[]` fields
- Enforced on POST and PUT
**Per-Field Image Dimensions** (checked from the decoded image header):
```yaml
fields:
- name: heroImage
type: file
validator:
accept: ["image/jpeg", "image/png", "image/webp"]
image:
minWidth: 1200
maxWidth: 3200
minHeight: 600
maxHeight: 2400
```
- Works for both `file` and `file[]` fields
- Applies to newly uploaded Data-URI files on POST and PUT
- Validates dimensions before the file is kept on disk
- Currently supported for dimension checks: PNG, JPEG, GIF, BMP, TIFF, WebP, and SVG
- `allowNonImages: true` lets a mixed-media field accept non-image uploads unchanged; image uploads on the same field are still checked against `minWidth`, `maxWidth`, `minHeight`, and `maxHeight`
- `allowUnknownDimensions: true` turns the validator into a best-effort check for `image/*`: uploads with unknown dimensions are accepted and min/max checks are skipped for that file
- HEIF and HEIC are currently detected as image MIME types, but are not yet supported by strict dimension checks; with `allowUnknownDimensions: true` they can still pass if the field accepts them
---
## Image Filters
Image filters allow on-the-fly image processing when serving files. Filters are configured per collection and applied via query parameter.
### Configuration
```yaml
collections:
- name: posts
imageFilter:
thumbnail:
- width: 200
height: 200
fit: true
quality: 80
outputType: webp
hero:
- width: 1200
height: 600
fill: true
anchor: center
quality: 85
```
### Usage
```
GET /api/v1/blog/posts/{id}/image/photo.jpg?filter=thumbnail
```
### Caching
Filtered images are cached on disk as:
```
{originalPath}.{filterName}.{extension}
```
Example: `photo.jpg` with filter `thumbnail` → `photo.thumbnail.webp`
Use `?no-cache=1` to force regeneration.
---
## FilterParams Reference
Each filter is an array of processing steps. All parameters are optional.
### Resize Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `width` | int | Target width in pixels |
| `height` | int | Target height in pixels |
| `fit` | bool | Resize to fit within dimensions (preserves aspect ratio, no crop) |
| `fill` | bool | Resize to fill dimensions (crops to fit, uses `anchor`) |
| `resampling` | string | Resampling algorithm (see below) |
| `anchor` | string | Crop anchor point for `fill` mode (see below) |
If neither `fit` nor `fill` is set, `imaging.Resize` is used (stretches to exact dimensions).
### Effect Parameters
| Parameter | Type | Range | Description |
|-----------|------|-------|-------------|
| `brightness` | float64 | -100 to 100 | Adjust brightness |
| `saturation` | float64 | -100 to 100 | Adjust saturation |
| `contrast` | float64 | -100 to 100 | Adjust contrast |
| `gamma` | float64 | > 0 | Gamma correction |
| `blur` | float64 | ≥ 0 | Gaussian blur sigma |
| `sharpen` | float64 | ≥ 0 | Unsharp mask sigma |
| `invert` | bool | — | Invert colors |
| `grayscale` | bool | — | Convert to grayscale |
### Output Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `quality` | int | JPEG/WebP quality (1-100) |
| `lossless` | bool | WebP lossless mode |
| `outputType` | string | Output format: `jpeg`, `png`, `webp`, `gif`, `bmp`, `tiff` |
| `skipLargerDimension` | bool | Skip processing if source is smaller than target |
| `skipLargerFilesize` | bool | Keep original if processed file would be larger |
### Resampling Algorithms
| Value | Algorithm | Quality/Speed |
|-------|-----------|---------------|
| `nearestNeighbor` | Nearest Neighbor | Fastest, lowest quality |
| `box` | Box | Fast, low quality |
| `linear` | Bilinear | Good balance |
| `hermite` | Hermite | Good quality |
| `mitchellNetravali` | Mitchell-Netravali | High quality |
| `catmullRom` | Catmull-Rom | High quality |
| `bSpline` | B-Spline | High quality, smooth |
| `gaussian` | Gaussian | High quality |
| `lanczos` | Lanczos | Highest quality, slowest |
| `bartlett` | Bartlett | Good quality |
| `hann` | Hann | Good quality |
| `hamming` | Hamming | Good quality |
| `blackman` | Blackman | High quality |
| `welch` | Welch | Good quality |
| `cosine` | Cosine | Good quality |
### Anchor Positions
Used with `fill: true` to determine crop position:
| Value | Position |
|-------|----------|
| `center` | Center (default) |
| `topLeft` | Top-left |
| `top` | Top-center |
| `topRight` | Top-right |
| `left` | Center-left |
| `right` | Center-right |
| `bottomLeft` | Bottom-left |
| `bottom` | Bottom-center |
| `bottomRight` | Bottom-right |
---
## Supported Image Formats
### Input Formats
| Format | Library |
|--------|---------|
| JPEG | `imaging.Open` |
| PNG | `imaging.Open` |
| GIF | `imaging.Open` |
| BMP | `imaging.Open` |
| TIFF | `imaging.Open` |
| WebP | `golang.org/x/image/webp` decoder |
### Output Formats
| Format | Library |
|--------|---------|
| JPEG | `imaging` |
| PNG | `imaging` |
| WebP | `chai2010/webp` encoder |
Auto-detect: If `outputType` is not set, format is determined from the target file extension.
---
## Processing Pipeline
When multiple filter steps are defined, they are applied **sequentially**:
```yaml
imageFilter:
optimized:
- width: 1200 # Step 1: Resize
height: 800
fit: true
sharpen: 0.5
- quality: 80 # Step 2: Compress
outputType: webp
```
Per step:
1. Skip check (if `skipLargerDimension` and source is smaller)
2. Resize (`fit`, `fill`, or plain resize)
3. Apply effects (blur, brightness, contrast, gamma, saturation, sharpen, invert, grayscale)
4. Encode to output format with quality settings
5. Skip check (if `skipLargerFilesize` and result is larger)
If the final file would be larger than the source and `skipLargerFilesize` is set, the original file is copied to the target path instead.
---
## Image Processing in Hooks
The `image.filter()` function provides the same processing capabilities in hooks:
```js
context.image.filter("/path/to/source.jpg", "/path/to/output.webp", [
{
width: 800,
height: 600,
fit: true,
quality: 85,
outputType: "webp"
}
]);
```
See [06-hooks.md](06-hooks.md#image--image-processing) for full API reference.
src/content/docs/server/09-llm-integration.md
+260
View File
@@ -0,0 +1,260 @@
---
title: 'LLM Integration'
---
# LLM Integration
## Overview
Tibi-Server provides a built-in LLM proxy that supports OpenAI-compatible APIs with streaming, token budgets, and usage logging. Users can interact with LLMs through a unified API that handles provider resolution, model validation, and quota management.
## Configuration
### Server Config
```yaml
llm:
providers:
- name: openai
type: openai # "openai" or "openai-compatible"
apiKey: sk-... # API key
baseURL: https://api.openai.com/v1 # Optional, defaults to OpenAI
defaultModel: gpt-4 # Fallback model
models: # Allowed models (whitelist)
- gpt-4
- gpt-4-turbo
- gpt-3.5-turbo
maxTokensPerRequest: 4096 # Token limit per request
- name: local-llama
type: openai-compatible
apiKey: ""
baseURL: http://localhost:11434/v1
defaultModel: llama3
models:
- llama3
- codellama
```
### API Key via Environment Variable
```yaml
llm:
providers:
- name: openai
type: openai
apiKey: ${OPENAI_API_KEY} # Resolved from environment
```
Convention: `LLM_<UPPERCASE_PROVIDER_NAME>_APIKEY` (e.g., `LLM_OPENAI_APIKEY`).
## API Endpoints
### Chat Completion
```text
POST /api/v1/llm/chat
X-Auth-Token: <jwt-token>
Content-Type: application/json
```
**Request:**
```json
{
"provider": "openai",
"model": "gpt-4",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Hello!" }
],
"stream": false,
"temperature": 0.7,
"maxTokens": 1000
}
```
**Response (non-streaming):**
```json
{
"id": "chatcmpl-abc123",
"model": "gpt-4",
"choices": [
{
"index": 0,
"message": { "role": "assistant", "content": "Hello! How can I help?" },
"finishReason": "stop"
}
],
"usage": {
"promptTokens": 25,
"completionTokens": 10,
"totalTokens": 35
}
}
```
### Streaming
```json
{
"provider": "openai",
"model": "gpt-4",
"messages": [{ "role": "user", "content": "Tell me a story." }],
"stream": true
}
```
**Response:** SSE stream with `Content-Type: text/event-stream`
```text
data: {"id":"chatcmpl-abc","model":"gpt-4","choices":[{"index":0,"delta":{"role":"assistant","content":"Once"},"finishReason":""}]}
data: {"id":"chatcmpl-abc","model":"gpt-4","choices":[{"index":0,"delta":{"content":" upon"},"finishReason":""}]}
data: {"id":"chatcmpl-abc","model":"gpt-4","choices":[{"index":0,"delta":{"content":" a"},"finishReason":""}]}
data: [DONE]
```
### List Providers
```text
GET /api/v1/llm/providers
X-Auth-Token: <jwt-token>
```
**Response:**
```json
[
{
"name": "openai",
"type": "openai",
"defaultModel": "gpt-4",
"models": ["gpt-4", "gpt-4-turbo", "gpt-3.5-turbo"]
},
{
"name": "local-llama",
"type": "openai-compatible",
"defaultModel": "llama3",
"models": ["llama3", "codellama"]
}
]
```
## Model Resolution
1. If `model` is provided in the request → validated against provider's `models` whitelist
2. If `model` is empty → provider's `defaultModel` is used
3. If the model is not in the whitelist → `400 Bad Request`
## Token Budget System
Administrators can assign per-provider token budgets to users:
```json
{
"username": "editor1",
"llmBudgets": [
{
"provider": "openai",
"tokenBudget": 100000,
"tokensUsed": 45230,
"periodStart": "2024-01-01T00:00:00Z",
"periodEnd": "2024-02-01T00:00:00Z"
}
]
}
```
### LLMBudget Structure
| Field | Type | Description |
| ------------- | ------ | ---------------------------------- |
| `provider` | string | Provider name |
| `tokenBudget` | int | Maximum tokens allowed in period |
| `tokensUsed` | int | Tokens consumed so far |
| `periodStart` | time | Budget period start |
| `periodEnd` | time | Budget period end |
### Budget Check Flow
1. Before each LLM request, **user budgets** are read from the database
2. `CheckBudget()` verifies:
- Budget period has not expired (`PeriodEnd` > now)
- Tokens used < token budget
3. If budget exceeded → `429 Too Many Requests`
4. If the user has a `primaryOrgId`, the **org's budget** is checked as well (soft check — only fails if the org has a budget for this provider and it's exhausted)
5. After completion, `RecordUsage()` increments `tokensUsed` on both the user document and the org document (if applicable)
### Org-Level Budget Cascade
Orgs can have their own `llmBudgets` (see [Orgs & Teams](18-orgs-teams.md#llm-budget-cascade)). When a user with a `primaryOrgId` makes an LLM request:
- **User budget** is checked first (hard check — must exist and have quota)
- **Org budget** is checked second via `user.primaryOrgId` (soft check — only enforced if configured)
- Usage is deducted from both budgets
- The `llm_usage` log entry includes both `userId` and `orgId`
This allows multiple users to share an org-wide token pool while each maintaining individual quotas.
### Token Estimation
When the upstream provider does not return usage information, tokens are estimated:
```go
// ~4 characters per token
func EstimateTokens(text string) int {
return len(text) / 4
}
// +4 tokens per message for framing overhead
func EstimateUsage(messages []ChatMessage, completionContent string) UsageInfo
```
## Usage Logging
Every LLM request is logged to the `llm_usage` collection in the system database:
```json
{
"id": "...",
"userId": "60a1b2c3d4...",
"orgId": "507f1f77bcf86cd799439011",
"provider": "openai",
"model": "gpt-4",
"promptTokens": 25,
"completionTokens": 10,
"totalTokens": 35,
"timestamp": "2024-01-15T10:30:00Z",
"insertTime": "2024-01-15T10:30:00Z"
}
```
The `orgId` field is populated from the user's `primaryOrgId` when present, linking usage to the billed org.
## Authentication
All LLM endpoints require a valid JWT token (`X-Auth-Token` header). Both admin and regular users can use LLM endpoints, but only admins can modify user `llmBudgets`.
## Provider Implementation
Tibi-Server uses the **OpenAI API format** as its universal protocol. All providers (including `openai-compatible`) implement the same interface:
```go
type Provider interface {
Name() string
ChatCompletion(ctx context.Context, req ChatRequest) (*ChatResponse, error)
ChatCompletionStream(ctx context.Context, req ChatRequest) (*ChatStream, error)
}
```
This means any OpenAI-compatible API (Ollama, vLLM, Azure OpenAI, LiteLLM, etc.) can be used by setting `type: openai-compatible` with the appropriate `baseURL`.
## Streaming Implementation Details
- Streaming uses `bufio.Scanner` for SSE parsing
- The `stream_options.include_usage = true` flag is set for OpenAI to get usage info in the final chunk
- `ChatStream.Read()` returns `io.EOF` when `data: [DONE]` is received
- Usage information from the final chunk is captured via `GetUsage()`
src/content/docs/server/10-audit.md
+277
View File
@@ -0,0 +1,277 @@
---
title: 'Audit Log'
---
# Audit Log
## Overview
Tibi-Server provides an audit logging system that records data changes (create, update, delete) with full snapshots. Audit logs are stored in the system database and queryable via API with role-based visibility.
## Configuration
### Server-Level Configuration
```yaml
audit:
enabled: true
defaultTTL: "720h" # Go duration (30 days). "0" = keep forever.
```
### Project-Level Configuration
```yaml
# in project config.yml (opt-in, default false for gradual migration)
audit:
enabled: true
```
> **Migration:** Audit is only active when **both** `server.audit.enabled` **and** `project.audit.enabled` are `true`. Existing projects without `audit.enabled` remain dark even when the server flag is enabled - add `audit.enabled: true` per project you want to migrate. Project level only supports `enabled` (no `actions`).
### Collection-Level Configuration
```yaml
collections:
- name: posts
audit:
enabled: true
actions:
- create
- update
- delete
```
**Default audited actions:** `create`, `update`, `delete`
**Available collection-level actions:** `create`, `update`, `delete`, `get`
**Available action-level actions (for `actions/*.yml`):** `post`, `get` (default `[post, get]`, `enabled` defaults to `true` if omitted)
> **Note:** `db.deleteMany` and `db.updateMany` are audited as `delete` and `update` respectively — all covered by the default actions.
>
> **Note:** Bulk API operations (`POST /collection` with array body, `PUT /collection?filter=...` and `DELETE /collection?filter=...`) are audited as `bulkCreate`, `bulkUpdate` and `bulkDelete` with the filter/count logged.
>
> **Note:** `login`, `reload`, and `shutdown` are system-level actions that are always logged when `audit.enabled` is `true`. They cannot be controlled via collection-level `audit.actions`.
### ShouldAudit Logic
An action is audited if:
1. Server-level `audit.enabled` is `true`
2. **Project-level `audit.enabled` is `true`** (opt-in, `nil`/`false` = off - `internal/models/auditlog.go:IsProjectAuditEnabled`)
3. Collection/Action-level `audit.enabled` is `true` (or not explicitly `false`)
3. The action is in the collection's `actions` list (or defaults are used)
## AuditLog Model
```json
{
"id": "...",
"insertTime": "2024-01-15T10:30:00Z",
"userId": "60a1b2c3d4...",
"username": "admin",
"userRole": 0,
"projectNamespace": "myblog",
"projectName": "My Blog",
"collection": "posts",
"action": "update",
"documentId": "60a1b2c3d4e5f6...",
"source": {
"type": "controller",
"collection": "",
"method": "PUT",
"step": "",
"file": "",
"line": 0
},
"snapshot": {
"title": "Old Title",
"body": "Old content..."
},
"changes": {
"title": "New Title"
},
"filter": null,
"count": null,
"ip": "192.168.1.1",
"authMethod": "jwt",
"tokenLabel": "",
"tokenPrefix": "",
"channel": "mcp"
}
```
### Authentication Context
Each audit entry includes authentication metadata:
| Field | Description | Values |
|-------|-------------|--------|
| `authMethod` | How the request was authenticated | `jwt`, `admin-token`, or empty |
| `tokenLabel` | Human-readable label from admin token config | e.g. `ci-deploy`, `monitoring` |
| `tokenPrefix` | First 8 characters of the admin token | e.g. `agent-to` |
| `userId` | User ID (only for JWT auth) | MongoDB ObjectId |
| `username` | Username (only for JWT auth) | e.g. `admin` |
| `ip` | Client IP address | e.g. `192.168.1.1` |
For JWT-authenticated requests, `userId`/`username`/`userRole` are populated and token fields are empty.
For admin token requests, `authMethod`/`tokenLabel`/`tokenPrefix` are populated and user fields are empty.
For PAT-authenticated requests (incl. MCP), `authMethod` is `access-token` with the token's name/prefix.
### Channel (REST vs. MCP)
| Field | Description | Values |
|-------|-------------|--------|
| `channel` | Through which interface the action came in | `mcp` for MCP tool calls, empty for REST (and older entries) |
Filter MCP-originated entries with `?filter={"channel":"mcp"}`. The concrete
tool follows from `collection` + `action` (`documentId` set = single, `filter`
set = bulk). See [MCP-Server](23-mcp.md).
### Source Types
| Type | Description |
| ------------ | ------------------------------------------ |
| `controller` | Direct API request (CRUD endpoint) |
| `hook` | Database operation from a hook script |
| `system` | Internal system operation |
| `job` | Database operation from a scheduled job |
For hook sources, additional fields are populated:
- `collection` — the hook's parent collection
- `method` — the HTTP method that triggered the hook
- `step` — hook step (e.g. `read`, `return`, `bind`, `validate`, `create`, `update`, `delete`)
- `file` — JavaScript source file
- `line` — line number in the JS file (from goja callstack)
## API
### Query Audit Logs
```text
GET /api/v1/audit
X-Auth-Token: <jwt-token>
```
**Query Parameters:**
| Parameter | Type | Default | Description |
| --------- | ----------- | -------------- | -------------------------------------------------------------------- |
| `filter` | JSON string | — | MongoDB-style filter (validated against AuditLog fields) |
| `limit` | int | 50 | Max results (capped at `maxLimit`, default max: 10,000) |
| `offset` | int | 0 | Pagination offset |
| `sort` | string | `-insertTime` | Sort field(s), prefix `-` for descending |
| `count` | string | — | `"1"` or `"true"` to include total count in `X-Results-Count` header |
**Configurable Limits:**
The audit endpoint enforces query result limits:
- **Default limit:** 50 (configurable via `audit.defaultLimit` in server config)
- **Maximum limit:** 10,000 (configurable via `audit.maxLimit` in server config)
- If `defaultLimit` exceeds `maxLimit`, it is automatically capped to `maxLimit`.
- See [Configuration](02-configuration.md) for YAML and environment variable options.
**Example:**
```bash
curl -H "X-Auth-Token: $TOKEN" \
'http://localhost:8080/api/v1/audit?filter={"collection":"posts","action":"update"}&limit=20&sort=-insertTime'
```
### Role-Based Visibility
**Admin (role=0):**
- Sees all audit entries across all projects
**User (role=1):**
- Only sees entries for projects and collections where they have GET permission
- System-sourced entries (`source.type: "system"`) are hidden
- Snapshot data is filtered through the collection's projection rules (same include/exclude logic as MongoDB projections)
### Sensitive Fields & audit.return Hooks
Audit snapshots store the **full document** at the time of the operation. Fields that are normally stripped by `get.return` hooks (e.g. passwords, API keys) are still present in audit snapshots.
To prevent sensitive data from being exposed through the audit endpoint, use the `audit.return` hook on the collection:
```yaml
hooks:
audit:
return:
type: javascript
file: hooks/users/audit_return.js
```
```js
// hooks/users/audit_return.js
;(function () {
var results = context.results
if (results && results.length > 0) {
if (results[0].snapshot) {
delete results[0].snapshot.password
delete results[0].snapshot.apiKey
}
if (results[0].changes) {
delete results[0].changes.password
delete results[0].changes.apiKey
}
}
return { results: results }
})()
```
Key properties:
- Runs for **all roles** including admin
- Affects existing snapshots retroactively (filtering happens at read-time)
- Each entry is processed individually (`context.one = true`, `context.results` has exactly one entry)
- Returning `{ results: [] }` suppresses the entry from the response
- Throwing an exception aborts the entire audit request
## Database Indexes
Four indexes are automatically created on the audit collection:
1. **TTL Index:** On `insertTime` — auto-deletes entries after configured TTL
2. **Project + Time:** `{projectNamespace, insertTime}`
3. **User + Time:** `{userId, insertTime}`
4. **Document:** `{collection, documentId, insertTime}`
## Write Behavior
- Audit log writes are **asynchronous by default** (fire-and-forget goroutine with own database session)
- Synchronous mode available via `AuditWriteSync = true` (used in tests)
- Each write uses its own MongoDB session to avoid blocking the main request
## What Gets Logged
### Controller Actions
| Action | Snapshot | Changes | Filter |
| -------- | ------------------------ | ---------------- | ------ |
| `create` | — | Created document | — |
| `update` | Document before update | Update data | — |
| `delete` | Document before deletion | — | — |
### Hook Database Actions
| Action | Snapshot | Changes | Filter |
| --------------------- | ------------------------ | ---------------- | ------------------- |
| `db.create` | — | Created document | — |
| `db.update` | Document before update | Update data | — |
| `db.delete` | Document before deletion | — | — |
| `db.deleteMany` | — | — | Filter used + count |
| `db.updateMany` | — | Update data | Filter used + count |
### Bulk API Actions
| Action | Snapshot | Changes | Filter |
| ------------ | -------- | ----------- | ------------------- |
| `bulkCreate` | — | — | Count + IDs |
| `bulkUpdate` | — | Update data | Filter used + count |
| `bulkDelete` | — | — | Filter used + count |
> `db.deleteMany` and `db.updateMany` are audited under the actions `delete` and `update` respectively. Bulk API actions (`bulkCreate`, `bulkUpdate`, `bulkDelete`) are also audited as `create`, `update` and `delete`.
src/content/docs/server/11-jobs.md
+154
View File
@@ -0,0 +1,154 @@
---
title: 'Jobs (Scheduled Tasks)'
---
# Jobs (Scheduled Tasks)
## Overview
Jobs are scheduled JavaScript programs that run on a cron schedule. They use the same goja runtime as hooks but execute independently of HTTP requests. Jobs are defined per project in the `config.yml`.
## Configuration
```yaml
jobs:
- cron: "0 */6 * * *" # Every 6 hours
type: javascript
file: jobs/cleanup.js
timeout: 30000 # Milliseconds (optional)
meta: # Arbitrary metadata (optional)
maxAge: 86400
- cron: "0 8 * * *" # Every day at 8:00
type: javascript
file: jobs/daily_report.js
timeout: 60000
```
### JobConfig Fields
| Field | Type | Required | Description |
| --------- | ------ | -------- | ------------------------------------------------------ |
| `cron` | string | yes | Cron expression (standard 5-field format) |
| `type` | string | yes | Job type (currently only `"javascript"`) |
| `file` | string | yes | Path to JavaScript file (relative to config directory) |
| `timeout` | int | no | Execution timeout in **milliseconds** |
| `meta` | object | no | Arbitrary metadata accessible via `config.project()` |
### Cron Expression Format
```text
┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-6, 0=Sunday)
│ │ │ │ │
* * * * *
```
Examples:
- `*/5 * * * *` — Every 5 minutes
- `0 * * * *` — Every hour
- `0 0 * * *` — Daily at midnight
- `0 8 * * 1-5` — Weekdays at 8:00
- `0 0 1 * *` — First of each month
## Available Context
Jobs have access to most hook context packages, with the following **exceptions**:
| Package | Available in Jobs? | Reason |
| ------------------- | ------------------ | ------------------------ |
| `db.*` | Yes | |
| `http.*` | Yes | |
| `fs.*` | Yes | |
| `smtp.*` | Yes | |
| `jwt.*` | Yes | |
| `bcrypt.*` | Yes | |
| `crypto.*` | Yes | |
| `base64.*` | Yes | |
| `json.*` | Yes | |
| `xml.*` | Yes | |
| `tpl.*` | Yes | |
| `image.*` | Yes | |
| `pdf.*` | Yes | |
| `exec.*` | Yes | |
| `config.*` | Yes | |
| `debug.*` | Yes | |
| `charset.*` | Yes | |
| `channel.send` | Yes | |
| `cookie.*` | **No** | No HTTP request/response |
| `user.*` | **No** | No HTTP request |
| `response.*` | **No** | No HTTP response |
| `request` | **No** | No HTTP request |
| `channel.subscribe` | **No** | No HTTP response for SSE |
## Job Examples
### Cleanup Old Documents
```js
// jobs/cleanup.js
var threshold = new Date();
threshold.setDate(threshold.getDate() - 30);
var result = db.deleteMany("temp_data", {
filter: {
createdAt: { "$lt": { "$date": threshold.toISOString() } }
}
});
context.debug.dump("Cleaned up " + result.removed + " documents");
```
### Daily Report Email
```js
// jobs/daily_report.js
var yesterday = new Date();
yesterday.setDate(yesterday.getDate() - 1);
var newPosts = db.count("posts", {
filter: {
insertTime: { "$gte": { "$date": yesterday.toISOString() } }
}
});
var newUsers = db.count("_users", {
filter: {
insertTime: { "$gte": { "$date": yesterday.toISOString() } }
}
});
var html = tpl.execute(
context.fs.readFile(context.fs.configDir() + "/templates/daily_report.html"),
{ newPosts: newPosts, newUsers: newUsers, date: yesterday.toISOString() }
);
context.smtp.sendMail({
to: "admin@example.com",
from: "reports@example.com",
subject: "Daily Report - " + yesterday.toISOString().split("T")[0],
html: html
});
```
### Notify via Realtime Channel
```js
// jobs/notify.js
context.channel.send("system", {
type: "scheduled_maintenance",
message: "Scheduled maintenance in 30 minutes",
timestamp: new Date().toISOString()
});
```
## Execution
- Jobs run in a `JOB_CONTEXT` (FunctionContextType = 2)
- Each job execution creates a fresh goja runtime
- Jobs execute under the project's configured UID (suexec)
- Job errors are logged server-side and reported to Sentry
- Audit log entries from job database operations have `source.type: "job"`
src/content/docs/server/12-deployment.md
+152
View File
@@ -0,0 +1,152 @@
---
title: 'Deployment'
---
# Deployment
## System Requirements
| Component | Requirement |
|-----------|-------------|
| **Go** | 1.23+ (toolchain 1.24) |
| **MongoDB** | 4.4+ (with replica set for transactions) |
| **OS** | Linux (required for suexec UID isolation) |
| **wkhtmltopdf** | Required for PDF generation (optional) |
## Docker Deployment
### Production Dockerfile
```dockerfile
# Multi-stage build
FROM golang:1.24 AS builder
WORKDIR /app
COPY . .
RUN go build -o server ./cmd/server
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates wkhtmltopdf && rm -rf /var/lib/apt/lists/*
COPY --from=builder /app/server /usr/local/bin/server
CMD ["server"]
```
### Docker Compose
```yaml
version: '3.8'
services:
tibi:
build: .
ports:
- "8080:8080"
volumes:
- ./config.yml:/app/config.yml
- ./projects:/app/projects
environment:
- API_MONGOURI=mongodb://mongo:27017
- API_JWTSECRET=your-jwt-secret-here
depends_on:
- mongo
mongo:
image: mongo:7
command: ["--replSet", "rs0"]
volumes:
- mongo-data:/data/db
ports:
- "27017:27017"
volumes:
mongo-data:
```
### Development with Air (Hot Reload)
A `Dockerfile.air` is provided for development with [Air](https://github.com/cosmtrek/air) hot-reloading:
```bash
docker compose -f docker-compose-local.yml up
```
## Configuration File
See [02-configuration.md](02-configuration.md) for the complete configuration reference.
Minimal `config.yml`:
```yaml
api:
port: 8080
mongoUri: mongodb://localhost:27017
jwtSecret: change-me-to-a-long-random-string
dbPrefix: tibi
projectsDir: ./projects
```
## UID/GID Isolation (suexec)
When running as **root**, tibi-server uses thread-level UID/GID isolation for multi-tenant security:
### How It Works
1. The main goroutine locks its OS thread at startup (`runtime.LockOSThread()`)
2. For each unique UID/GID pair, a dedicated goroutine is spawned that:
- Locks its OS thread
- Changes thread credentials via `SYS_SETGID` / `SYS_SETUID` syscalls
- Waits for jobs in an event loop
3. File operations and `exec.command` calls are dispatched to the appropriate UID thread
4. Thread pool scales up to `runtime.NumCPU()` per UID/GID pair
### Thread Pool Behavior
- New threads are spawned on 1-second send timeout (pool is busy)
- Maximum threads per UID/GID = `runtime.NumCPU()`
- Threads persist for the lifetime of the server
### File-Based UID Detection
`RunInFilesUID(filename)` determines the UID/GID from `os.Stat()` of the given file. This is used for project data directories — the filesystem owner determines the execution context.
### When Not Root
When running as a non-root user, suexec is not available. All file operations run under the server process's UID.
## Graceful Shutdown
The server supports graceful shutdown via:
1. **OS Signal:** `SIGINT` or `SIGTERM`
2. **Admin API:** `POST /api/v1/admin/shutdown` with admin token (requires `server.shutdown` permission)
Shutdown process:
1. HTTP server stops accepting new connections
2. In-flight requests are allowed to complete (with timeout)
3. All realtime hub instances are closed (wakes blocking subscriptions)
4. MongoDB connections are closed
## Admin Reload
```
POST /api/v1/_/{project}/_/admin/reload
X-Admin-Token: your-admin-token
```
Reloads a specific project's configuration from disk without restarting the server. This:
- Re-reads all `config.yml` files from `projectsDir`
- Recompiles all hooks and jobs
- Rebuilds collection indexes
- Does **not** drop existing realtime connections (they continue on the old config until they reconnect)
## Health Check
```
GET /api/v1/version
```
Returns server version information. No authentication required. Suitable for load balancer health checks.
## Monitoring
- **Prometheus Metrics:** `GET /metrics` — see [13-openapi-metrics.md](13-openapi-metrics.md)
- **Sentry Integration:** Configure `api.sentry` in `config.yml` for error tracking
- **Audit Logs:** Enable `audit.enabled: true` for data change tracking
src/content/docs/server/13-openapi-metrics.md
+151
View File
@@ -0,0 +1,151 @@
---
title: 'OpenAPI & Metrics'
---
# OpenAPI & Metrics
## OpenAPI Specification
Tibi-Server auto-generates an OpenAPI 3.0.0 specification for each project based on its collection configuration.
### Endpoint
```
GET /api/v1/{namespace}/openapi
```
No authentication required.
### Query Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `lang` | `en` | Language for i18n fields (resolves `{en: "...", de: "..."}` maps) |
### Generated Content
The specification includes:
1. **Info:** Project name/namespace as title, version `"0.0.0"`
2. **Paths:** Two paths per collection:
- `/{collection}` — List endpoint (GET)
- `/{collection}/{id}` — Single document endpoint (GET)
3. **Schemas:** Generated from collection field definitions (recursive for nested objects)
4. **Security Schemes:** API key schemes derived from token-based permissions
### Collection Metadata for OpenAPI
Collections can customize their OpenAPI representation via `meta.openapi`:
```yaml
collections:
- name: posts
meta:
openapi:
disabled: false # Exclude from spec entirely
get:
disabled: false # Exclude list endpoint
summary: "List posts"
description: "Returns all published posts"
getOne:
disabled: false # Exclude single endpoint
summary: "Get post"
description: "Returns a single post by ID"
```
### Type Mapping
| Collection Field Type | OpenAPI Type |
|----------------------|--------------|
| `string` | `string` |
| `int` | `integer` |
| `float` | `number` |
| `bool` | `boolean` |
| `date` | `string` (format: date-time) |
| `object` | `object` (with nested properties) |
| `file` | `object` with `{ src: string }` |
| `string[]` | `array` of `string` |
| (other arrays) | `array` of corresponding type |
### i18n Support
Field names, descriptions, and summaries can be i18n maps:
```yaml
fields:
- name:
en: Title
de: Titel
key: title
type: string
```
The `lang` query parameter determines which language string is used in the generated spec.
---
## Prometheus Metrics
### Endpoint
```
GET /metrics
```
Registered on the root router (not under `/api/v1/`). Standard Prometheus text format. No authentication required.
### Exposed Metrics
#### `tibi_request_duration_seconds` (Summary)
Tracks response time for all HTTP handlers.
| Label | Description | Example |
|-------|-------------|---------|
| `method` | HTTP method | `GET`, `POST`, `PUT`, `DELETE` |
| `status` | HTTP status code | `200`, `404`, `500` |
| `base` | System collection name (non-project routes) | `user`, `project`, `login` |
| `ns` | Project namespace (project routes) | `myblog`, `myshop` |
| `col` | Collection name | `posts`, `products` |
| `id` | Document ID indicator | `1` (with ID) or `""` (list) |
**Quantiles:**
| Quantile | Error Margin |
|----------|-------------|
| p50 | ±0.05 |
| p75 | ±0.025 |
| p90 | ±0.01 |
| p99 | ±0.001 |
### Example Output
```
# HELP tibi_request_duration_seconds Histogram of response time for handler in seconds
# TYPE tibi_request_duration_seconds summary
tibi_request_duration_seconds{method="GET",status="200",base="",ns="myblog",col="posts",id="",quantile="0.5"} 0.002
tibi_request_duration_seconds{method="GET",status="200",base="",ns="myblog",col="posts",id="",quantile="0.9"} 0.005
tibi_request_duration_seconds{method="GET",status="200",base="",ns="myblog",col="posts",id="",quantile="0.99"} 0.012
tibi_request_duration_seconds_sum{method="GET",status="200",base="",ns="myblog",col="posts",id=""} 1.234
tibi_request_duration_seconds_count{method="GET",status="200",base="",ns="myblog",col="posts",id=""} 500
```
### Grafana Dashboard Example
Query for average request duration per endpoint:
```promql
rate(tibi_request_duration_seconds_sum[5m]) / rate(tibi_request_duration_seconds_count[5m])
```
Query for p99 latency per collection:
```promql
tibi_request_duration_seconds{quantile="0.99", ns="myblog"}
```
### Integration
The metrics middleware (`measureResponseDuration`) is registered globally on all routes. It measures the duration of each request and records it with the appropriate labels.
Standard Go runtime metrics are also available via the default Prometheus registry (`go_*` metrics).
src/content/docs/server/14-security.md
+636
View File
@@ -0,0 +1,636 @@
---
title: 'Security Audit — Tibi-Server'
---
# Security Audit — Tibi-Server
> **Audit-Datum:** 27. Februar 2026
> **Scope:** Vollständiges Security-Audit des tibi-server Quellcodes
> **Methode:** Statische Code-Analyse aller sicherheitsrelevanten Komponenten
---
## Zusammenfassung
| Schweregrad | Anzahl | Status |
|-------------|--------|--------|
| **KRITISCH** | 3 | Offen |
| **HOCH** | 6 | Offen |
| **MITTEL** | 7 | Offen |
| **NIEDRIG** | 5 | Offen / Teilweise gelöst |
| **Gelöst** | 1 | ✅ |
---
## Security Architecture
### Authentication Layer
| Mechanism | Scope | Header/Cookie |
|-----------|-------|---------------|
| JWT (HMAC-SHA256) | User authentication | `X-Auth-Token` header |
| Refresh Token | Token renewal | `tibi-refresh-token` cookie (HttpOnly, SameSite=Strict) |
| Admin Tokens | Server-level CRUD, management & namespace collection access | `X-Admin-Token` header or `Authorization: Bearer` |
| Token Permissions | API key access | Configurable header/query parameter |
### Authorization Layers
1. **Role-Based:** Admin (role=0) bypasses all checks
2. **Project-Based:** Users assigned to projects via `project.users[]`
3. **Collection-Based:** Per-collection permission sets with method restrictions
4. **Pre-Filters:** Automatic query filters based on permission context
5. **Projection Restrictions:** Limit visible fields per permission level
### Data Isolation
- **Database:** Each project gets its own MongoDB database (`{prefix}_{namespace}`)
- **Filesystem:** suexec UID/GID isolation (Linux, requires root)
- **Realtime:** Each project has its own Hub instance
### Input Validation
- Collection field validation (type, required, allowZero, in, eval)
- MongoDB filter operator whitelist (dangerous operators blocked)
- File path sanitization (`CleanFilePath`)
- User input validation (username alphanumeric, password min 3 chars)
### Bulk Operation Security
Bulk operations (`POST` with JSON array body, `PUT`/`DELETE` without `:id`) require **two** permission checks:
1. The standard method permission must be granted (`post`, `put` or `delete` with `allow: true`)
2. The `bulk` flag must be set to `true` in the object form of the permission
Method permissions support two formats:
- **Boolean shorthand:** `post: true` — allows single-document POST only (no bulk)
- **Object form:** `post: {allow: true, bulk: true}` — allows both single and bulk POST
This applies equally to PUT and DELETE. Without explicit `bulk: true`, bulk endpoints return **403 Forbidden**. This ensures that existing projects are not automatically exposed to bulk operations when upgrading.
---
## Findings
### CRITICAL
#### ~~C1. MD5 Password Hashing~~ ✅ BEHOBEN (2025-02-27)
**Location:** `internal/models/login.go`, `internal/models/user.go`
**CVSS-Schätzung:** 9.1 (Kritisch) → **behoben**
**Description:** User passwords wurden zuvor mit MD5 gehasht, was kryptographisch gebrochen ist.
**Fix:** Passwort-Hashing wurde auf bcrypt (`golang.org/x/crypto/bcrypt`, DefaultCost) umgestellt mit abwärtskompatibler MD5-Migration:
- `user.go:ValidateData()` — neue Passwörter werden mit bcrypt gehasht
- `login.go:LoginCheck()` — Login prüft per `CheckPassword()` sowohl bcrypt als auch legacy-MD5
- Transparente Migration: bei erfolgreichem Login mit MD5-Hash wird automatisch auf bcrypt re-gehasht und in der DB aktualisiert
- Neue Hilfsfunktionen: `IsBcryptHash()`, `CheckPassword()` (exportiert für Tests)
- Unit-Tests: `TestIsBcryptHash`, `TestCheckPassword` + alle bestehenden Tests grün
---
#### C2. No Rate Limiting on Login Endpoint — ✅ BEHOBEN
**Location:** `internal/ratelimit/ratelimit.go`, `internal/controller/ratelimit_middleware.go`, `internal/controller/setup.go`
**CVSS-Schätzung:** 8.8 (Hoch–Kritisch)
**Description:** The `POST /api/v1/login` endpoint has no rate limiting or account lockout mechanism. Combined with MD5 password hashing and a minimum password length of only 3 characters, this makes the system highly vulnerable to brute-force attacks.
**Impact:** An attacker can attempt unlimited login attempts without any delay, lockout, or CAPTCHA. With a 3-character minimum password, the search space is trivially small.
**Lösung implementiert:**
- In-Memory Rate Limiter mit exponentiellem Backoff (1s → 2s → 4s → ... → 60s Cap)
- Duales Tracking: per IP und per Username (getrennt)
- Automatischer Reset nach 15 Minuten Inaktivität
- HTTP 429 + `Retry-After` Header bei Blockierung
- Konfigurierbar via `config.yml` (`rateLimit`-Sektion) und Environment-Variablen
- Standardmäßig aktiviert, kann via `rateLimit.enabled: false` deaktiviert werden
- Hintergrund-Cleanup-Goroutine für abgelaufene Einträge
---
#### C3. Admin Token Comparison Vulnerable to Timing Attacks — ✅ BEHOBEN
**Location:** `internal/config/admin_token.go`, `internal/controller/admin_reload.go`
**CVSS-Schätzung:** 7.5 (Hoch) → **behoben**
**Description:** Admin tokens wurden mit Go's `==` Operator verglichen, was nicht constant-time ist und Timing-Angriffe ermöglicht.
**Lösung implementiert:**
- Alle 3 Token-Vergleichsstellen auf `crypto/subtle.ConstantTimeCompare()` umgestellt:
- `config/admin_token.go:CheckAdminToken()` — Server-Admin-Token-Prüfung
- `config/admin_token.go:GetAdminTokenInfo()` — Admin-Token-Metadaten-Abfrage
- `controller/admin_reload.go:ProjectReload()` — Projekt-Admin-Token-Prüfung
- Constant-Time-Vergleich verhindert Rückschlüsse auf Token-Inhalte durch Zeitmessung
---
### HIGH
#### H1. Secrets in Plain Text Configuration
**Location:** `config.yml`, `internal/config/server.go`
**CVSS-Schätzung:** 7.5
**Description:** JWT secrets, SMTP passwords, and LLM API keys are stored in plain text in the configuration file. The shipped `config.yml` contains a hardcoded JWT secret:
```yaml
api:
jwtSecret: 237uiejakymxeou3zhakN # Hardcoded production secret in repo!
```
**Impact:** Anyone with read access to the repository or config file can forge JWT tokens, access SMTP credentials, and use LLM API keys.
**Mitigation:** Environment variable substitution is available (`${ENV_VAR}` syntax).
**Recommendation:**
- Replace all hardcoded secrets in `config.yml` with env-var references
- Warn on startup if `jwtSecret` does not use `${...}` syntax
- Add `.gitignore` entry for production configs
- Consider adding a vault integration
---
#### H2. Token Authentication via Query Parameters — ℹ️ AKZEPTIERTES RISIKO
**Location:** `internal/models/permission.go`
**CVSS-Schätzung:** 7.0
**Status:** Akzeptiertes Risiko (bewusste Design-Entscheidung) + Timing-Attack-Fix
**Description:** Token-based permissions check both headers and query parameters. Query parameters appear in server logs, browser history, referrer headers, and proxy logs, exposing authentication tokens.
**Bewertung:** Die Query-Parameter-Unterstützung ist ein bewusstes Feature für Anwendungsfälle, in denen Header nicht gesetzt werden können (z.B. direkte URL-Aufrufe, Webhooks). Das Risiko ist dokumentiert und dem Betreiber bekannt.
**Timing-Attack-Fix (2025-02-27):** Token-Vergleiche in `GetTokenPermissionSet()` wurden von `==` auf `subtle.ConstantTimeCompare()` umgestellt, um Timing-Angriffe auf Token-Werte zu verhindern.
**Empfehlung für Betreiber:**
- Token-Werte über Environment-Variablen konfigurieren (`"x-api-key:${API_KEY}"`)
- Proxy/Loadbalancer so konfigurieren, dass Query-Parameter nicht geloggt werden
- Wo möglich, Header-basierte Token-Übergabe bevorzugen
---
#### H3. Refresh Token Cookie Missing Secure Flag — ✅ BEHOBEN
**Location:** `internal/models/login.go`, `internal/config/server.go`
**CVSS-Schätzung:** 7.0 → **behoben**
**Description:** The refresh token cookie had `Secure: false` hardcoded.
**Lösung implementiert:**
- Neuer Config-Eintrag `api.secureCookies` (bool, Default: `true`)
- Environment-Variable: `API_SECURE_COOKIES`
- Beide `SetCookie`-Aufrufe (Login + Logout) verwenden jetzt `config.Server.API.IsSecureCookies()`
- Default `true` → Cookies werden nur über HTTPS gesendet
- Für lokale Entwicklung (HTTP) kann `secureCookies: false` gesetzt werden
---
#### H5. SSRF via http.fetch in Hooks
**Location:** `internal/hook/context_http.go`
**CVSS-Schätzung:** 6.5
**Description:** The `http.fetch` and `http.fetchStream` functions in hooks allow requests to arbitrary URLs without restriction. A hook can fetch internal network resources (e.g. `http://169.254.169.254/` for cloud metadata, `http://localhost:27017/` for MongoDB).
```go
request, err := http.NewRequest(method, url, reqBody)
// No URL validation — any scheme, any host, any port
```
**Impact:** Server-Side Request Forgery (SSRF) allowing access to internal services, cloud metadata endpoints, and potentially the MongoDB instance.
**Recommendation:**
- Implement an optional URL whitelist/blocklist at config level
- Block requests to private IP ranges (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.0.0/16`, `127.0.0.0/8`) by default
- Document SSRF risks for project administrators
---
#### H6. Unrestricted Shell Command Execution in Hooks
**Location:** `internal/hook/context_exec.go`
**CVSS-Schätzung:** 6.5
**Description:** The `exec.command` function executes any binary without path or name restrictions. While arguments are passed as separate `args[]` (safe from shell injection), the command name is controlled by hook code:
```go
cmd := exec.CommandContext(timeoutCtx, command, cmdArgs...)
```
The `dir` option also allows setting arbitrary working directories:
```go
if dir, ok := options["dir"].(string); ok {
cmd.Dir = dir
}
```
**Mitigation:** Commands execute under project UID via suexec isolation. Default dir is project config directory.
**Impact:** Arbitrary code execution on the host with project-level privileges. Without suexec (non-root deployment), commands run as the server process user.
**Recommendation:**
- Implement a command whitelist at the project config level
- Restrict `dir` option through `CleanFilePath` validation
- Log all command executions for audit purposes
---
### MEDIUM
#### M1. Error Message Information Disclosure
**Location:** `internal/utils/error.go`, `internal/dbapi/helper.go`
**CVSS-Schätzung:** 5.3
**Description:** By default (`config.response.error.message` not set), the raw error message is included in API responses:
```go
errorInResponse := errorString
if config.Server.Response.Error.Message != "" {
errorInResponse = config.Server.Response.Error.Message
}
```
When `response.error.stack: true` is set, Go stack traces are exposed:
```go
if config.Server.Response.Error.Stack {
o["errorStack"] = errorStack
}
```
**Impact:** Detailed error messages can reveal internal paths, database structure, Go type names, and stack traces.
**Recommendation:**
- Default to generic error messages in production
- Only enable detailed errors when explicitly configured for development
- Never expose stack traces in production
---
#### M2. File Upload: No Size Limit or Type Validation
**Location:** `internal/controller/collection.go:75-140`
**CVSS-Schätzung:** 5.3
**Description:** File uploads via base64-encoded data in JSON have no size limit or content type validation:
```go
if i := strings.Index(srcPrefix, ","); i > 0 &&
strings.HasPrefix(srcPrefix, "data:") &&
strings.Index(srcPrefix, "base64") > 0 { // validate as data
```
The MIME type is extracted from the `data:` URL but not validated against an allowlist. Any file type can be uploaded.
**Impact:**
- Denial of service via extremely large base64 payloads (decoded in memory)
- Storage exhaustion attacks
- Upload of executable files or malicious content
**Recommendation:**
- Implement configurable maximum file size limits
- Add optional MIME type allowlists per collection field
- Consider limiting request body size at the middleware level (Gin's `MaxMultipartMemory`)
---
#### M3. File Upload: Filename Not Sanitized
**Location:** `internal/controller/collection.go:119-121`
**CVSS-Schätzung:** 5.3
**Description:** The original filename from the upload payload is used directly in the file path:
```go
if filename, ok := fileObj["path"].(string); ok {
path += "/" + filename
}
```
While the `path` is within the upload directory tree, the `filename` value is not sanitized. A filename like `../../etc/cron.d/malicious` combined with upload path could potentially escape the intended directory.
**Mitigation:** The upload path structure `uploadPath + "/" + id + "/" + field` provides some containment. The ID is server-generated.
**Recommendation:**
- Sanitize filenames: strip directory separators, use `filepath.Base()`, restrict to alphanumeric + allowed extensions
- Apply `CleanFilePath` validation to constructed upload paths
---
#### M4. ReDoS Risk via $regex Filter Operator
**Location:** `internal/dbapi/filter_valid_functions.go:28`
**CVSS-Schätzung:** 5.3
**Description:** The `$regex` operator is allowed in user-provided query filters:
```go
"$regex": true,
```
MongoDB's `$regex` uses PCRE, which is vulnerable to catastrophic backtracking with malicious patterns. An attacker can craft regex patterns that cause exponential CPU usage on the database server.
**Impact:** Denial of service via CPU exhaustion on MongoDB.
**Recommendation:**
- Set a timeout for MongoDB read operations
- Consider restricting `$regex` to authenticated users only
- Add regex complexity validation (max length, disallow nested quantifiers)
- Document ReDoS risks
---
#### M5. Prometheus /metrics Endpoint Unauthenticated
**Location:** `internal/controller/setup.go:135`
**CVSS-Schätzung:** 5.0
**Description:** The Prometheus metrics endpoint is publicly accessible without authentication:
```go
r.GET("/metrics", gin.WrapH(promhttp.Handler()))
```
**Impact:** Exposes internal metrics including request rates, error rates, latency distributions, and potentially Go runtime metrics, providing reconnaissance data to attackers.
**Recommendation:**
- Add authentication (admin token or basic auth) to `/metrics`
- Or restrict access via network-level controls and document this requirement
---
#### M6. No Request Timeout Middleware
**Location:** `internal/controller/setup.go`
**CVSS-Schätzung:** 5.0
**Description:** A TODO exists for request timeout middleware that was never implemented:
```go
// TODO: timeout middleware
```
**Impact:** Long-running requests (e.g. complex queries, large file uploads, slow hooks) can tie up server goroutines indefinitely, enabling resource exhaustion attacks.
**Recommendation:** Implement `http.TimeoutHandler` or Gin timeout middleware with configurable duration (e.g. 30s default, configurable per collection).
---
#### M7. JWT Token Contains Full User Object
**Location:** `internal/models/login.go:95-103`
**CVSS-Schätzung:** 4.3
**Description:** The JWT access token includes the full user object (minus password), including arbitrary `meta` data:
```go
userCopy := *user
userCopy.InsertTime = nil
userCopy.UpdateTime = nil
userCopy.Meta = nil
tokenData = &userCopy
```
The `Meta` is set to `nil`, but `Permissions`, `LLMBudgets`, and other fields are included. The token is not encrypted, only signed.
**Impact:** Token leakage exposes user metadata and internal permission structures. Tokens are also long-lived (60 minutes) without revocation support.
**Recommendation:**
- Minimize token claims to `{id, username, role}`
- Load full user data from DB when needed
- Consider shorter token lifetime with more frequent refresh
- Implement token revocation (e.g. via token blacklist or `iat` validation)
---
### LOW
#### L1. Typo in Log Output (Unchanged)
**Location:** `internal/controller/setup.go:62`
**Description:** Log message reads `"senty gin activated"` instead of `"sentry gin activated"`.
---
#### L2. Weak Password Policy
**Location:** `internal/models/user.go:38`
**CVSS-Schätzung:** 3.7
**Description:** Password minimum length is 3 characters:
```go
Password *string `json:"password,omitempty" bson:"password" validator:"required,min=3"`
```
**Impact:** Extremely weak passwords are permitted, compounding the brute-force risk.
**Recommendation:** Increase minimum to 8+ characters. Consider complexity requirements.
---
#### L3. `allowUpperPaths: true` in Default Config
**Location:** `config.yml:15`
**Description:** The default config allows traversal above config directory:
```yaml
security:
allowAbsolutePaths: false
allowUpperPaths: true
```
This enables `fs.readFile("../../etc/passwd")` from hooks (though constrained by suexec UID).
**Recommendation:** Default to `false` for `allowUpperPaths` and only enable when explicitly needed.
---
#### L4. Hook Cookie Functions Without Scope Restrictions
**Location:** `internal/hook/context_cookie.go`
**Description:** The `cookie.set` function in hooks can set cookies with arbitrary names, paths, domains, and security flags. A malicious hook could overwrite authentication cookies or set cookies for different domains.
**Recommendation:** Document this as a trust boundary — hook code has the same trust level as server-side code. Consider restricting cookie domain to the current request's domain.
---
#### L5. LLM Quota Race Condition
**Location:** `internal/llm/quota.go:61-104`
**Description:** LLM budget checking and recording happens in two separate operations (read, then write). Under concurrent requests, a user could exceed their token budget before the write propagates:
```go
// 1. Check budget (read)
if err := llm.CheckBudget(budgets, req.Provider); err != nil { ... }
// 2. Use tokens (API call happens here)
// 3. Record usage (write)
_ = qm.RecordUsage(auth.ID, req.Provider, resp.Model, usage, budgets)
```
**Impact:** Minor over-budget usage under high concurrency; not exploitable for significant financial impact.
**Recommendation:** Accept current behavior with documentation, or use MongoDB atomic `$inc` for budget tracking.
---
### RESOLVED
#### ~~R1. Commented-Out Collection Count Endpoint~~
**Location:** `internal/controller/setup.go`, `internal/controller/collection_count.go`
The dedicated `GET /_/:project/:collection/_count` endpoint is now implemented with full permission pipeline.
---
## Security Posture Assessment
### Strengths ✅
1. **MongoDB Operator Whitelist:** Filter operators are validated against an explicit allowlist (`filter_valid_functions.go`), blocking dangerous operators like `$where`
2. **Suexec Process Isolation:** Projects run with dedicated UID/GID on Linux, providing filesystem-level sandboxing
3. **Permission System Depth:** Multi-layered permission model (public → token → user → custom) with pre-filters and projection restrictions
4. **Hook Timeout Support:** JS hooks have configurable timeouts to prevent infinite loops
5. **Audit Logging:** Comprehensive audit trail for admin actions with token identification
6. **Bulk Operation Guard:** Requires explicit `bulk: true` config, preventing accidental mass operations
7. **File Path Sanitization:** `CleanFilePath` provides directory traversal protection with configurable strictness
8. **Error Code Tracking:** Each error gets a unique code for correlation without exposing internals (when `response.error.message` is configured)
9. **Env-Var Substitution:** Secrets can be externalized via `${ENV_VAR}` syntax
10. **Admin Token Granularity:** Fine-grained permission system with wildcards for server admin tokens
### Weaknesses ❌
1. **Cryptographic:** ~~MD5 password hashing~~ ✅ behoben (bcrypt), no salt ✅ behoben (bcrypt nutzt eingebauten Salt), weak password policy
2. **Network Security:** No TLS termination (expected to be behind proxy), ~~no rate limiting~~ ✅ behoben, ~~Cookie Secure=false~~ ✅ behoben (konfigurierbar, Default: true)
3. **Token Security:** ~~Timing-attack-vulnerable comparisons~~ ✅ behoben (constant-time compare), query parameter leakage ℹ️ akzeptiertes Risiko, no revocation
4. **Input Validation Gaps:** No file upload size limits, filename not sanitized, ReDoS via `$regex`
5. **Execution Safety:** Unrestricted shell commands and HTTP fetch in hooks (SSRF)
6. **Missing Middleware:** No request timeout, ~~no rate limiting~~ ✅ behoben (Login-Rate-Limiting), no body size limits
---
## Prioritized Remediation Plan
### Phase 1 — Critical (Immediate, 1-2 Weeks)
| # | Finding | Effort | Impact |
|---|---------|--------|--------|
| ~~C1~~ | ~~Replace MD5 with bcrypt + migration path~~ | ~~Medium~~ | ✅ **ERLEDIGT** — bcrypt + MD5-Migration implementiert |
| ~~C2~~ | ~~Add login rate limiting~~ | ~~Low~~ | ✅ **ERLEDIGT** — Exponentielles Backoff Rate Limiting (IP + Username) implementiert |
| ~~C3~~ | ~~Use `subtle.ConstantTimeCompare` for tokens~~ | ~~Low~~ | ✅ **ERLEDIGT** — Alle Token-Vergleiche auf constant-time umgestellt |
| H1 | Remove hardcoded JWT secret from config.yml | Low | Prevents token forgery from source |
### Phase 2 — High (2-4 Weeks)
| # | Finding | Effort | Impact |
|---|---------|--------|--------|
| H3 | Make cookie `Secure` flag configurable | Low | ✅ **ERLEDIGT** — `api.secureCookies` Config-Eintrag (Default: true) |
| L2 | Increase minimum password length to 8 | Low | Reduces brute-force surface |
| M1 | Default to generic error messages | Low | Prevents info disclosure |
| M6 | Implement request timeout middleware | Medium | Prevents resource exhaustion |
### Phase 3 — Medium (1-2 Months)
| # | Finding | Effort | Impact |
|---|---------|--------|--------|
| ~~H2~~ | ~~Deprecate query-parameter tokens~~ | ~~Medium~~ | ℹ️ **AKZEPTIERTES RISIKO** — bewusste Design-Entscheidung, Timing-Fix angewendet |
| H5 | Add SSRF protection for http.fetch | Medium | Blocks internal network access |
| H6 | Add exec command whitelist config | Medium | Limits command execution |
| M2 | Add file upload size/type validation | Medium | Prevents resource exhaustion |
| M3 | Sanitize uploaded filenames | Low | Prevents path manipulation |
| M4 | Add regex complexity limits | Medium | Prevents ReDoS on MongoDB |
| M5 | Protect /metrics endpoint | Low | Hides internal metrics |
| M7 | Minimize JWT token claims | Medium | Reduces exposure surface |
### Phase 4 — Hardening (Ongoing)
| # | Finding | Effort | Impact |
|---|---------|--------|--------|
| L3 | Default `allowUpperPaths` to false | Low | Tighter default security |
| L4 | Document hook trust boundaries | Low | Operational awareness |
| L5 | Atomic LLM budget tracking | Low | Prevents minor over-use |
---
## Security Best Practices
### Configuration
```yaml
api:
jwtSecret: ${JWT_SECRET} # Always use env vars for secrets
cors:
allowOrigins:
- https://yourdomain.com # Never use "*" in production
allowCredentials: true
security:
allowAbsolutePaths: false
allowUpperPaths: false # Restrict to config directory
response:
error:
message: "An error occurred" # Generic error for production
stack: false # Never in production
```
### Permissions
```yaml
permissions:
public:
methods:
get: true # Minimal public access
filter:
published: true # Always pre-filter public data
user:
methods:
get: true
post: true
put: true
filter:
author:
$subst:
expr: "auth.id" # Users only see own documents
validProjections:
- summary # Restrict visible fields
```
### Token Permissions
```yaml
permissions:
"x-api-key:${API_KEY}": # Use env vars for token values
methods:
get: true
filter:
public: true
```
### Hooks Security
```js
// Always validate input
var req = request();
var body = json.parse(req.body());
if (!body.email || body.email.indexOf("@") === -1) {
throw { status: 400, json: { error: "invalid_email" } };
}
// Use bcrypt for password hashing in custom auth flows
var hash = bcrypt.hash(body.password, { cost: 12 });
// Sanitize exec commands
var allowedCommands = ["convert", "ffmpeg", "python3"];
if (allowedCommands.indexOf(command) === -1) {
throw { status: 400, json: { error: "command_not_allowed" } };
}
```
src/content/docs/server/15-troubleshooting.md
+232
View File
@@ -0,0 +1,232 @@
---
title: 'Troubleshooting & FAQ'
---
# Troubleshooting & FAQ
## Common Errors
### Connection Errors
**Error:** `failed to connect to MongoDB`
- Check that MongoDB is running and reachable
- Verify `api.mongoUri` in `config.yml` or `API_MONGOURI` env var
- For Docker: ensure the MongoDB container is on the same network
- For replica sets: ensure all members are reachable
**Error:** `server selection timeout`
- MongoDB replica set may not be initialized
- Run: `rs.initiate()` in the MongoDB shell
---
### Authentication Errors
**Error:** `401 Unauthorized` on all requests
- Check that `X-Auth-Token` header is set correctly
- Verify JWT is not expired (60 min default)
- Ensure `api.jwtSecret` matches between token generation and validation
**Error:** `403 Forbidden` on collection access
- Check collection permissions in project `config.yml`
- Verify user is assigned to the project (`project.users[]`)
- Check if token-based permission has correct value (wrong value = immediate deny)
- Admin role (0) bypasses all permission checks
**Error:** Login returns wrong password even though it's correct
- Passwords are stored as MD5 hashes
- Check if password was stored in plaintext vs. hashed
- Use `crypto.md5("password")` in a hook to verify the expected hash
---
### Hook Errors
**Error:** `ReferenceError: require is not defined`
- Hooks run in goja (ES5.1+), not Node.js
- There is no `require()`, `import`, or npm
- Use the built-in context packages (`db`, `http`, `fs`, etc.)
**Error:** `ReferenceError: async is not defined`
- goja does not support `async/await`
- All hook functions are synchronous
- Use `http.fetchStream` for streaming, not async patterns
**Error:** `TypeError: X is not a function`
- Check the function name — it may have been renamed
- See the [legacy compatibility table](06-hooks.md#legacy-compatibility)
- Common: `readCollection` → `db.find`, `mail` → `smtp.sendMail`
**Error:** JS hook timeout
- Increase timeout in `hookConfig.timeout` (milliseconds)
- For streaming hooks, use `channel.subscribe()` which auto-disables timeout
- Check for infinite loops in hook code
**Error:** Hook stopped after client disconnected
- By default, hooks follow the incoming HTTP request lifecycle
- Set `ignoreRequestCancel: true` on the hook config only for critical hooks that must continue after client disconnect
- `ignoreRequestCancel` does not disable the hook timeout; increase `timeout` separately if needed
- `DisableTimeout()` only removes the execution timeout for specific built-in long-running operations; it does not, by itself, ignore request cancellation
**Error:** Hook exception not returning expected format
- Ensure the thrown object has the right keys: `status`, `json`, `html`, `string`, `bytes`, `file`
- If `response.write()` was already called, exceptions can only abort — no new response body
---
### File Upload Errors
**Error:** Files not being saved
- Check that the upload path exists and is writable
- File fields must use Base64 data URI format: `data:mime/type;base64,...`
- Field type must be `file` or `file[]` in collection config
**Error:** Image filter returns 404
- Verify the filter name exists in `imageFilter` config
- Check file path: `/{namespace}/{collection}/{id}/{field}/{filename}?filter={name}`
- Use `?no-cache=1` to force regeneration
---
### Configuration Errors
**Error:** Project not loading
- Check YAML syntax in project `config.yml`
- Verify the project directory is under `api.projectsDir`
- Check file permissions (readable by server process)
- Use admin reload: `POST /api/v1/_/{project}/_/admin/reload`
**Error:** Environment variable not resolved
- Format must be `${VAR_NAME}` (with dollar and braces)
- Variable must be set in the server's environment
- Env file path must be correct in `config.yml`
**Error:** CORS not working
- CORS is configured at three levels: server → project → collection
- Check if `merge: true` is set for inheriting parent CORS
- `allowOrigins: ["*"]` does not work with `allowCredentials: true`
- Note: `exposeHeaders` always replaces (not appends) even with `merge: true` (known bug)
---
## Debugging Tools
### Server-Side Debugging
```js
// Print to stdout
context.debug.dump(context.data, "some label");
// Get Sentry trace ID for error tracking
var traceId = debug.sentryTraceId();
// Pause execution
context.debug.sleep(5000); // 5 seconds
```
### Request Inspection
```js
var req = request();
context.debug.dump({
method: req.method,
path: req.path,
url: req.url,
host: req.host,
clientIp: req.clientIp,
authHeader: req.header("Authorization"),
body: req.body()
});
```
### Database Debugging
```js
// Check what's in a collection
var all = db.find("posts", { limit: 5 });
context.debug.dump(all);
// Count documents
var count = db.count("posts", {
filter: { status: "published" }
});
context.debug.dump("Published posts:", count);
```
### HTTP Debugging
Use the `test.http` file in the project root for quick API testing with VS Code's REST Client extension.
---
## FAQ
### Can I use TypeScript in hooks?
No. Hooks run in goja (ECMAScript 5.1+). However, the `tibi-types` package provides TypeScript definitions for IDE autocompletion during development.
### Can I use async/await?
No. The goja runtime is synchronous. Use `http.fetchStream` for streaming and `channel.subscribe` for real-time patterns.
### How do I handle file uploads?
Embed files as Base64 data URIs in the JSON body. Set the field type to `file` or `file[]`. The server automatically extracts and saves the file, replacing the URI with a file path.
### How do I serve static files?
Configure `assets` in your project config with a name and path, then access via `/api/v1/{namespace}/assets/{name}/{path}`.
### Can I use WebSockets?
No. Tibi-Server uses Server-Sent Events (SSE) for real-time communication. Use `channel.subscribe` in a hook for the server side and `EventSource` on the client.
### How do I run database migrations?
Use hooks or jobs. Create a `post.before` hook on a migration collection, or a cron job that checks and applies necessary schema changes.
### How do I reset a user's password?
As admin, update the user via PUT. The password will be automatically MD5-hashed on save.
### How do I add custom endpoints?
Use hooks with custom response writing. A `get.before` hook that calls `response.write()` and throws effectively creates a custom endpoint.
### What happens if a hook throws without a status?
Default status is `500`. The error message is included in a JSON error response.
### How do I keep a critical hook running after the client disconnects?
Set `ignoreRequestCancel: true` on that hook and configure a suitable `timeout`.
Use this only for truly critical side effects such as order submission or writing to an outbox. For larger workflows, prefer persisting work and continuing asynchronously in a job instead of holding the HTTP request open.
### How do I debug CORS issues?
1. Check browser DevTools → Network → look for preflight (OPTIONS) requests
2. Verify `allowOrigins` includes your domain (exact match, not regex)
3. Check if `allowCredentials` is needed (cookies, auth headers)
4. Verify `allowMethods` includes the HTTP method you're using
5. Remember: wildcard origin `*` does not work with credentials
### How do I scale horizontally?
Currently, tibi-server is designed for single-instance deployment. Realtime channels are in-memory and not synchronized across instances. For multi-instance setups, put a load balancer in front and ensure sticky sessions for SSE connections.
src/content/docs/server/16-cookbook.md
+686
View File
@@ -0,0 +1,686 @@
---
title: 'Collection Config Cookbook'
---
# Collection Config Cookbook
Ready-to-use collection configuration recipes for common use cases.
---
## Blog with Drafts and Publishing
```yaml
collections:
- name: posts
fields:
- key: title
type: string
validation:
required: true
- key: slug
type: string
validation:
required: true
- key: body
type: string
- key: excerpt
type: string
- key: coverImage
type: file
- key: tags
type: string[]
- key: author
type: string
- key: authorName
type: string
- key: status
type: string
validation:
required: true
in:
- draft
- published
- archived
- key: publishedAt
type: date
permissions:
public:
methods:
get: true
filter:
status: published
user:
methods:
get: true
post: true
put: true
delete: true
projections:
summary:
title: 1
slug: 1
excerpt: 1
coverImage: 1
tags: 1
authorName: 1
publishedAt: 1
full:
title: 1
slug: 1
body: 1
excerpt: 1
coverImage: 1
tags: 1
author: 1
authorName: 1
status: 1
publishedAt: 1
hooks:
post.before: hooks/posts/before_create.js
imageFilter:
thumbnail:
- width: 400
height: 300
fit: true
quality: 80
outputType: webp
hero:
- width: 1200
height: 600
fill: true
anchor: center
quality: 85
outputType: webp
indexes:
- keys:
slug: 1
options:
unique: true
- keys:
status: 1
publishedAt: -1
audit:
enabled: true
```
**Hook: `hooks/posts/before_create.js`**
```js
var auth = user.auth();
context.data.author = auth.id;
context.data.authorName = auth.username;
if (!context.data.status) {
context.data.status = "draft";
}
if (context.data.status === "published" && !context.data.publishedAt) {
context.data.publishedAt = new Date().toISOString();
}
```
---
## User Profiles (Self-Managed)
```yaml
collections:
- name: profiles
fields:
- key: userId
type: string
validation:
required: true
- key: displayName
type: string
- key: bio
type: string
- key: avatar
type: file
- key: website
type: string
- key: social
type: object
fields:
- key: twitter
type: string
- key: github
type: string
permissions:
public:
methods:
get: true
user:
methods:
get: true
post: true
put: true
filter:
userId:
$subst:
expr: "auth.id"
projections:
public:
displayName: 1
bio: 1
avatar: 1
website: 1
social: 1
indexes:
- keys:
userId: 1
options:
unique: true
imageFilter:
small:
- width: 100
height: 100
fill: true
anchor: center
quality: 80
outputType: webp
```
---
## E-Commerce Products with Variants
```yaml
collections:
- name: products
fields:
- key: name
type: string
validation:
required: true
- key: description
type: string
- key: price
type: float
validation:
required: true
- key: currency
type: string
validation:
in: [EUR, USD, GBP]
- key: images
type: file[]
- key: category
type: string
- key: tags
type: string[]
- key: variants
type: object[]
fields:
- key: name
type: string
- key: sku
type: string
- key: price
type: float
- key: stock
type: int
- key: active
type: bool
permissions:
public:
methods:
get: true
filter:
active: true
"x-api-key:${SHOP_API_KEY}":
methods:
get: true
post: true
put: true
delete: true
projections:
catalog:
name: 1
price: 1
currency: 1
images: 1
category: 1
tags: 1
indexes:
- keys:
category: 1
- keys:
tags: 1
- keys:
active: 1
category: 1
imageFilter:
thumb:
- width: 300
height: 300
fit: true
quality: 85
outputType: webp
large:
- width: 1200
height: 1200
fit: true
quality: 90
```
---
## Chat Messages with Realtime
```yaml
collections:
- name: messages
fields:
- key: room
type: string
validation:
required: true
- key: sender
type: string
- key: senderName
type: string
- key: text
type: string
validation:
required: true
- key: type
type: string
validation:
in: [text, image, system]
permissions:
user:
methods:
get: true
post: true
hooks:
post.before: hooks/messages/before_send.js
post.after: hooks/messages/after_send.js
indexes:
- keys:
room: 1
insertTime: -1
- name: chat-stream
hooks:
get.before: hooks/messages/stream.js
permissions:
user:
methods:
get: true
```
**Hook: `hooks/messages/before_send.js`**
```js
var auth = user.auth();
context.data.sender = auth.id;
context.data.senderName = auth.username;
if (!context.data.type) {
context.data.type = "text";
}
```
**Hook: `hooks/messages/after_send.js`**
```js
context.channel.send("chat-" + context.data.room, {
id: context.data.id,
sender: context.data.sender,
senderName: context.data.senderName,
text: context.data.text,
type: context.data.type,
time: context.data.insertTime
});
```
**Hook: `hooks/messages/stream.js`**
```js
var req = request();
var room = req.query("room");
if (!room) {
throw { status: 400, json: { error: "room parameter required" } };
}
context.response.header("Content-Type", "text/event-stream");
context.response.header("Cache-Control", "no-cache");
context.response.writeHeader(200);
var sub = channel.subscribe("chat-" + room, {
bufferSize: 200,
lastN: 50
});
var msg;
while ((msg = sub.receive()) !== null) {
context.response.writeSSE("message", context.json.stringify(msg));
}
sub.close();
throw { status: 200 };
```
---
## File Storage with Metadata
```yaml
collections:
- name: documents
fields:
- key: title
type: string
validation:
required: true
- key: file
type: file
validation:
required: true
- key: mimeType
type: string
- key: size
type: int
- key: uploadedBy
type: string
- key: folder
type: string
- key: tags
type: string[]
permissions:
user:
methods:
get: true
post: true
put: true
delete: true
hooks:
post.before: hooks/documents/before_upload.js
indexes:
- keys:
folder: 1
- keys:
tags: 1
- keys:
uploadedBy: 1
```
**Hook: `hooks/documents/before_upload.js`**
```js
var auth = user.auth();
context.data.uploadedBy = auth.id;
// Extract mime type from data URI
if (context.data.file && context.data.file.indexOf("data:") === 0) {
var mime = context.data.file.split(";")[0].split(":")[1];
context.data.mimeType = mime;
}
```
---
## Cross-Referenced Collections
```yaml
collections:
- name: authors
fields:
- key: name
type: string
validation:
required: true
- key: email
type: string
- key: avatar
type: file
- name: articles
fields:
- key: title
type: string
validation:
required: true
- key: body
type: string
- key: authorId
type: string
validation:
required: true
- key: categoryIds
type: string[]
hooks:
get.after: hooks/articles/after_get.js
- name: categories
fields:
- key: name
type: string
- key: slug
type: string
```
**Hook: `hooks/articles/after_get.js`**
```js
// Resolve author references
var authorIds = [];
for (var i = 0; i < context.result.length; i++) {
if (context.result[i].authorId && authorIds.indexOf(context.result[i].authorId) === -1) {
authorIds.push(context.result[i].authorId);
}
}
if (authorIds.length > 0) {
var authors = db.find("authors", {
filter: { id: { "$in": authorIds } },
selector: { name: 1, avatar: 1 }
});
var authorMap = {};
for (var j = 0; j < authors.length; j++) {
authorMap[authors[j].id] = authors[j];
}
for (var k = 0; k < context.result.length; k++) {
context.result[k].author = authorMap[context.result[k].authorId] || null;
}
}
```
---
## Event Calendar
```yaml
collections:
- name: events
fields:
- key: title
type: string
validation:
required: true
- key: description
type: string
- key: location
type: string
- key: startDate
type: date
validation:
required: true
- key: endDate
type: date
- key: allDay
type: bool
- key: recurring
type: string
validation:
in: ["", daily, weekly, monthly, yearly]
- key: image
type: file
- key: organizer
type: string
- key: maxParticipants
type: int
- key: currentParticipants
type: int
permissions:
public:
methods:
get: true
filter:
startDate:
$gte:
$subst:
eval: "new Date().toISOString()"
hooks:
put.before: hooks/events/register.js
indexes:
- keys:
startDate: 1
- keys:
location: 1
```
---
## Settings / Key-Value Store
```yaml
collections:
- name: settings
fields:
- key: key
type: string
validation:
required: true
- key: value
type: string
- key: group
type: string
permissions:
"x-admin-token:${SETTINGS_TOKEN}":
methods:
get: true
post: true
put: true
delete: true
user:
methods:
get: true
indexes:
- keys:
key: 1
options:
unique: true
- keys:
group: 1
```
---
## Bulk Operations
### Bulk Update — Change Category for Multiple Documents
Update all products in category "old" to category "new":
```bash
curl -X PUT \
"https://example.com/api/v1/_/myproject/products?filter={\"category\":\"old\"}" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"category": "new"}'
```
### Bulk Update — Increase Prices by 10%
Use the `$mul` operator to increase all prices by 10%:
```bash
curl -X PUT \
"https://example.com/api/v1/_/myproject/products?filter={\"category\":\"premium\"}" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"price": {"$mul": 1.1}}'
```
### Bulk Update — Decrement Stock
Use `$inc` with a negative value to reduce stock for all matched items:
```bash
curl -X PUT \
"https://example.com/api/v1/_/myproject/products?filter={\"warehouse\":\"A\"}" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"stock": {"$inc": -5}}'
```
### Bulk Update — Remove a Field
Use `$unset` to remove a field from all matched documents:
```bash
curl -X PUT \
"https://example.com/api/v1/_/myproject/products?filter={\"deprecated\":true}" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"legacyField": {"$unset": true}}'
```
### Bulk Delete — Delete by Filter
Delete all archived posts:
```bash
curl -X DELETE \
"https://example.com/api/v1/_/myproject/posts?filter={\"status\":\"archived\"}" \
-H "Authorization: Bearer $TOKEN"
```
### Optimised Bulk Update with Hook
For large datasets, use a `bulkUpdate` hook step with `context.db.updateMany()` for a single database call instead of per-document processing:
```yaml
collections:
- name: products
hooks:
put.bulkUpdate: hooks/products/bulk_update.js
put.return: hooks/products/bulk_return.js
```
**hooks/products/bulk_update.js:**
```javascript
// Optimised: single DB call to update all matching documents
var filter = context.filter;
var body = context.body;
var result = context.db.updateMany("products", filter, { changes: body });
context.log("Updated " + result.modifiedCount + " documents");
```
### Optimised Bulk Update with Operators in Hook
Use the `update` parameter instead of `changes` to pass raw MongoDB update operators:
```javascript
// Increase price by 10% for all matching products
var filter = context.filter;
var result = context.db.updateMany("products", filter, {
update: { "$mul": { "price": 1.1 } }
});
context.log("Updated " + result.modifiedCount + " documents");
```
### Optimised Bulk Delete with Hook
```yaml
collections:
- name: products
hooks:
delete.bulkDelete: hooks/products/bulk_delete.js
delete.return: hooks/products/bulk_return.js
```
**hooks/products/bulk_delete.js:**
```javascript
var filter = context.filter;
var result = context.db.deleteMany("products", filter);
context.log("Deleted " + result.deletedCount + " documents");
```
src/content/docs/server/17-field-level-permissions.md
+550
View File
@@ -0,0 +1,550 @@
---
title: 'Field-Level Permissions'
---
# Field-Level Permissions
## Überblick
Field-Level Permissions erlauben den Zugriff auf **einzelne Felder** innerhalb einer Collection einzuschränken. Das System verwendet ein **3-Layer-Kaskadenmodell** mit **Fehler-Modus** (400 Bad Request) statt stillem Stripping.
Zusätzlich können `readonly` und `hidden` auf Field-Definition-Ebene als **Eval-Ausdrücke** (JavaScript) definiert werden, die **pro Dokument** dynamisch ausgewertet werden.
**Zwei Feld-Typen:**
| Typ | POST/PUT | GET | Config |
|-----|----------|-----|--------|
| `readonlyFields` | **400-Fehler** wenn im Body | Normal sichtbar | Felder bleiben in `fields[]`; Liste in `yourPermissions` |
| `hiddenFields` | **400-Fehler** wenn im Body | Aus Response **gestripped** | Felder aus `fields[]` **entfernt**; Liste nicht in `yourPermissions` |
**Admin-Bypass:** Admin (role=0) und Admin-Token → `GetEffectivePermissionSet()` gibt `nil` zurück → keine Einschränkungen.
---
## Konfiguration
### Drei Ebenen (Kaskade)
```yaml
name: orders
# Ebene 1: Collection-Level (Basis für alle PermissionSets)
readonlyFields: [createdBy]
hiddenFields: [costPrice]
permissions:
user:
methods:
get: true
post: true
put: true
# Ebene 2: PermissionSet-Level (addiert/subtrahiert von Collection-Level)
readonlyFields: [customerName]
hiddenFields: [internalMargin]
fields:
- name: alwaysReadonly
type: number
readonly: true # Ebene 3: Field-Definition (absolutes Override)
- name: alwaysHidden
type: string
hidden: true # Ebene 3: Field-Definition (absolutes Override)
- name: neverReadonly
type: string
readonly: false # Ebene 3: entfernt Feld aus JEDEM effektiven readonly-Set
- name: conditionalReadonly
type: string
readonly:
eval: "$.status === 'confirmed'" # Ebene 3: Eval — readonly nur wenn Bedingung zutrifft
- name: conditionalHidden
type: string
hidden:
eval: "$.status === 'confirmed'" # Ebene 3: Eval — hidden nur wenn Bedingung zutrifft
```
### Kaskadenmodell im Detail
Die effektive Feldliste wird in 4 Schritten berechnet (`ResolveEffectiveFields`):
1. **Collection-Level** → Basis-Set: `readonlyFields: [createdBy]`
2. **PermissionSet-Level** → Hinzufügen/Entfernen mit `-` Prefix-Negation:
- `readonlyFields: [customerName]` → fügt `customerName` hinzu
- `readonlyFields: [-createdBy]` → entfernt `createdBy` aus dem Set
3. **Parent-Expansion** → Elternfelder werden auf alle Sub-Felder expandiert (z.B. `payment` → `payment.method`, `payment.cardNumber`)
4. **Field-Definition Override** (letztes Wort):
- `readonly: true` → Feld wird **immer** hinzugefügt (egal ob in Liste oder nicht)
- `readonly: false` → Feld wird **immer** entfernt (überschreibt alles)
- `hidden: true` / `hidden: false` → analog für Hidden-Felder
- `readonly: {eval: "..."}` / `hidden: {eval: "..."}` → **Eval-Ausdruck**, wird pro Dokument ausgewertet (siehe [Eval-Ausdrücke](#eval-ausdrücke-dynamische-feld-permissions))
### Ergebnis-Beispiel
Mit der obigen Config ergibt sich:
| Feld | user effektiv readonly | user effektiv hidden |
|------|----------------------|---------------------|
| createdBy | ✅ (Collection-Level) | — |
| customerName | ✅ (PermissionSet) | — |
| alwaysReadonly | ✅ (Field-Def `readonly: true`) | — |
| neverReadonly | ❌ (Field-Def `readonly: false` überschreibt) | — |
| costPrice | — | ✅ (Collection-Level) |
| internalMargin | — | ✅ (PermissionSet) |
| alwaysHidden | — | ✅ (Field-Def `hidden: true`) |
### Verschachtelung (Dot-Notation)
Felder in verschachtelten Objekten werden per Dot-Notation adressiert:
```yaml
hiddenFields:
- payment.cardNumber
- payment.cvv
readonlyFields:
- address.zip
```
Wenn ein **Elternfeld** gelistet wird, werden automatisch **alle Sub-Felder** mit einbezogen (es sei denn, ein Sub-Feld hat ein Field-Def-Override `false`).
### Token-basierte Permissions
Tokens erhalten ein eigenes PermissionSet und unterliegen denselben Feld-Einschränkungen:
```yaml
permissions:
"token:MYTOKEN":
methods:
get: true
post: true
readonlyFields: [status]
# Erbt costPrice von Collection-Level hiddenFields
```
---
## Eval-Ausdrücke (Dynamische Feld-Permissions)
### Überblick
Neben statischem `readonly: true/false` und `hidden: true/false` kann auf Field-Definition-Ebene ein **JavaScript-Eval-Ausdruck** definiert werden. Dieser wird **pro Dokument** ausgewertet und ermöglicht konditionale Feld-Permissions basierend auf Dokumentinhalt, User-Kontext oder HTTP-Methode.
**Wichtig:** Eval-Ausdrücke nehmen **nicht** an der statischen Kaskade (Layer 1–3) teil. Sie werden als **separater Post-Pass** nach den statischen Checks ausgeführt.
### Syntax
```yaml
fields:
- name: confirmedAt
type: string
readonly:
eval: "$.status === 'confirmed'" # JS-Ausdruck → readonly wenn true
- name: secretNote
type: string
hidden:
eval: "$.status === 'confirmed'" # JS-Ausdruck → hidden wenn true
```
### Verfügbare Kontext-Variablen
| Variable | Typ | Beschreibung |
|----------|-----|-------------|
| `$` | `object` | Das vollständige Dokument (bei POST: der Body; bei PUT: das bestehende DB-Dokument + Body-Merge; bei GET: das DB-Dokument) |
| `$this` | `any` | Der Wert des aktuellen Felds im Dokument |
| `$auth` | `object` | Der authentifizierte User (mit `id`, `username`, `role`, `permissions`) |
| `$method` | `string` | Die HTTP-Methode (`POST`, `PUT`, `GET`) |
| `$project` | `string` | Der Projektname |
| `$namespace` | `string` | Der API-Namespace |
### Beispiele
```yaml
# Feld readonly nach Bestätigung
readonly:
eval: "$.status === 'confirmed'"
# Feld nur für bestimmte User hidden
hidden:
eval: "$auth.role > 1"
# Feld readonly bei PUT, aber nicht bei POST
readonly:
eval: "$method === 'PUT'"
# Kombination: Feld readonly wenn Betrag > 1000
readonly:
eval: "$.amount > 1000"
```
### Zwei-Phasen-Architektur
1. **Phase 1 (Statisch):** Die 3-Layer-Kaskade wird normal aufgelöst. Eval-Felder werden bei der Kaskade **ignoriert** (`.BoolValue()` gibt `nil` zurück).
2. **Phase 2 (Eval):** Nach den statischen Checks wird pro Dokument der JS-Ausdruck evaluiert. Ergibt er `true`, wird das Feld wie ein statisches readonly/hidden-Feld behandelt (400 bei POST/PUT, Strip bei GET).
### Vorkompilierung
Eval-Ausdrücke werden beim Laden der Config (in `AddCollection`) zu `goja.Program`-Objekten vorkompiliert. Syntaxfehler werden beim Config-Reload als Error geloggt. Zur Laufzeit wird nur noch `prog.Run()` aufgerufen.
### Admin-Bypass
Admin-User (role=0) und Admin-Tokens umgehen **alle** Eval-Checks. `GetEffectivePermissionSet()` gibt `nil` zurück → der gesamte Eval-Block wird übersprungen.
### Verhalten pro Endpoint
| Endpoint | Eval-Verhalten |
|----------|---------------|
| **POST** | Body wird als Dokument-Kontext (`$`) verwendet. Eval-readonly/hidden → 400 |
| **PUT** | Bestehendes DB-Dokument als Kontext. Eval-readonly/hidden → 400 (inkl. Operator-Bodies) |
| **GET** | Pro Ergebnis-Dokument: Eval-hidden → Feld wird aus Response gestripped |
| **Bulk-POST** | Pro Zeile individuelle Eval-Auswertung |
| **Bulk-Update** | Eval-Auswertung vor Batch-Operation |
### Config-Auslieferung
Eval-Definitionen (`readonly: {eval: "..."}`, `hidden: {eval: "..."}`) werden als Teil der Field-Definitionen in `fields[]` mit ausgeliefert. Der Client kann die Eval-Ausdrücke direkt aus der Config lesen und clientseitig vorvalidieren (da nur JS unterstützt wird, kann der gleiche Ausdruck im Browser ausgeführt werden).
Für **Non-Admins** werden statisch hidden Feld-Definitionen wie gewohnt aus `fields[]` entfernt. Eval-hidden-Felder bleiben jedoch sichtbar, da sie nur per Dokumentkontext ausgewertet werden können.
---
## Verhalten pro Endpoint
### POST / Bulk-POST
```
POST /api/v1/_/:project/:collection
```
Nach Body-Parsing, **vor** Validierung und Hooks:
1. `GetEffectivePermissionSet()` bestimmen (admin → `nil` → kein Check)
2. Effektive `readonlyFields` berechnen (3-Layer-Kaskade)
3. `CheckFieldsPresent(body, readonlyFields)` → bei Treffer: **400** `{"error": "readonly fields must not be sent", "fields": [...]}`
4. Effektive `hiddenFields` berechnen
5. `CheckFieldsPresent(body, hiddenFields)` → bei Treffer: **400** `{"error": "hidden fields must not be sent", "fields": [...]}`
6. **Eval-Phase:** `EvalFieldReadonly(body, ...)` + `EvalFieldHidden(body, ...)` → bei Treffer: **400** analog
Bei Bulk-POST: Check wird **pro Zeile** ausgeführt. Fehlgeschlagene Zeilen werden mit Index im Error-Array zurückgegeben, erfolgreiche Zeilen werden trotzdem verarbeitet.
### PUT / Bulk-Update
```
PUT /api/v1/_/:project/:collection/:id
```
Wie POST, zusätzlich werden **Operator-Felder** geprüft (`$set`, `$inc`, `$mul`, `$unset`, etc.):
```json
{"$set": {"customerName": "New"}, "$inc": {"price": 10}}
```
→ Prüfung ob `customerName` oder `price` in readonly/hidden ist. Felder in `$set`, `$inc`, `$mul`, `$unset`, `$min`, `$max`, `$rename`, `$push`, `$pull`, `$addToSet` werden alle geprüft.
Zusätzlich: **Eval-Phase** wie bei POST, das bestehende DB-Dokument dient als Eval-Kontext (`$`).
### GET
```
GET /api/v1/_/:project/:collection
GET /api/v1/_/:project/:collection/:id
```
Nach DB-Read, nach Lookups, **vor** Return-Hook:
- Hidden-Felder werden aus jedem Result-Dokument **gestripped** (`StripFields`)
- **Eval-Hidden:** Pro Dokument wird `EvalFieldHidden()` ausgewertet; bei `true` wird das Feld zusätzlich gestripped
- Readonly-Felder bleiben **sichtbar** (nur Schreibschutz, kein Leseschutz)
- Gilt für Einzel- und Listen-Abfragen
### Config-Auslieferung (GET /api/v1/project/:id)
Die `enrichData()`-Funktion in `project.go` liefert rollenabhängig unterschiedliche Configs:
**Nicht-Admin (User, Public, Token):**
- `yourPermissions[collection].readonlyFields` = effektive readonly-Liste
- `yourPermissions[collection].hiddenFields` wird **nicht** ausgeliefert
- Hidden-Feld-Definitionen werden aus `fields[]` **entfernt** (Frontend sieht sie nicht)
- Eval-Definitionen bleiben in `fields[]` sichtbar (auch eval-hidden, da nur per Dokument auswertbar)
- `hiddenFields`-Arrays werden aus Collection-Config und PermissionSets **entfernt**
**Admin:**
- `yourPermissions[collection].readonlyFields` wird **nicht** ausgeliefert (keine Einschränkungen)
- `yourPermissions[collection].hiddenFields` = nur als **Hint** (z.B. für UI-Markierung)
- Alle Feld-Definitionen bleiben vollständig in `fields[]`
**Lazy-Copy-Optimierung:** Die shared API-Struct wird nur kopiert wenn tatsächlich Hidden-Felder zu strippen sind (`apiCopied`-Flag). Admin-Requests erzeugen keine Kopie. So wird Mutation des geteilten API-Objekts verhindert.
---
## Vollständiges Config-Beispiel
```yaml
name: orders
defaultLanguage: german
# Collection-Level (Ebene 1): Basis für alle PermissionSets
readonlyFields: [createdBy]
hiddenFields: [costPrice]
permissions:
public:
methods:
get: true
post: false
put: false
delete: false
readonlyFields: [status, customerName]
hiddenFields: [internalMargin, costPrice, payment.cardNumber]
user:
methods:
get: true
post: true
put: true
delete: false
readonlyFields: [customerName, neverReadonly]
hiddenFields: [internalMargin]
fulfillment:
methods:
get: true
put: true
readonlyFields: [customerName, price]
# kein hiddenFields → erbt nur Collection-Level costPrice
"token:APITOKEN":
methods:
get: true
post: true
readonlyFields: [status]
fields:
- name: title
type: string
validator:
required: true
- name: status
type: string
- name: customerName
type: string
- name: price
type: number
- name: internalMargin
type: number
- name: costPrice
type: number
- name: createdBy
type: string
- name: alwaysReadonly
type: number
readonly: true # Field-Def: immer readonly für alle nicht-Admins
- name: alwaysHidden
type: string
hidden: true # Field-Def: immer hidden für alle nicht-Admins
- name: neverReadonly
type: string
readonly: false # Field-Def: NIEMALS readonly (überschreibt PermSet-Einträge)
- name: payment
type: object
subFields:
- name: method
type: string
- name: cardNumber
type: string
```
### Ergebnis-Matrix (nach Kaskade)
| Feld | admin | user | public | token:APITOKEN | fulfillment |
|------|-------|------|--------|----------------|-------------|
| title | read+write | read+write | read | read+write | read+write |
| status | read+write | read+write | **readonly** | **readonly** | read+write |
| customerName | read+write | **readonly** | **readonly** | read+write | **readonly** |
| price | read+write | read+write | read | read+write | **readonly** |
| createdBy | read+write | **readonly** | **readonly** | **readonly** | **readonly** |
| alwaysReadonly | read+write | **readonly** | **readonly** | **readonly** | **readonly** |
| neverReadonly | read+write | read+write | read+write | read+write | read+write |
| internalMargin | read+write | **hidden** | **hidden** | read+write | read+write |
| costPrice | read+write | **hidden** | **hidden** | **hidden** | **hidden** |
| alwaysHidden | read+write | **hidden** | **hidden** | **hidden** | **hidden** |
| payment.cardNumber | read+write | read+write | **hidden** | read+write | read+write |
---
## Implementierung
### Dateien
| Datei | Zweck |
|-------|-------|
| `internal/utils/field_filter.go` | Generische Kaskaden-Engine, Check/Strip-Funktionen |
| `internal/models/field_permissions.go` | Model-spezifische Wrapper, Eval-Funktionen (Eval/Resolve/Collect) |
| `internal/models/collection.go` | Struct-Definitionen (BoolOrEval, ReadonlyFields, HiddenFields) |
| `internal/models/api.go` | Config-Laden, `compileFieldEvals()` (JS-Vorkompilierung) |
| `internal/models/permission.go` | `GetEffectivePermissionSet()` — Admin-Bypass-Logik |
| `internal/models/project.go` | `enrichData()` — Config-Auslieferung mit Lazy-Copy + Eval-Felder |
| `internal/controller/collection_post.go` | POST-Enforcement (statisch + eval) |
| `internal/controller/collection_put.go` | PUT-Enforcement (statisch + eval, inkl. Operator-Check) |
| `internal/controller/collection_get.go` | GET Hidden-Stripping (statisch + eval pro Dokument) |
| `internal/controller/collection_bulk_post.go` | Bulk-POST per-row Enforcement (statisch + eval) |
| `internal/controller/collection_bulk_update.go` | Bulk-Update Enforcement (statisch + eval) |
### Structs
```go
// PermissionSet (collection.go)
type PermissionSet struct {
Methods map[string]MethodPermission
Filter dbcrud.M
ValidProjections []string
ReadonlyFields []string // Felder die für dieses PermSet readonly sind
HiddenFields []string // Felder die für dieses PermSet hidden sind
CompiledFilterSubs []*utils.CompiledSubst
}
// CollectionConfig (collection.go)
type CollectionConfig struct {
Name string
ReadonlyFields []string // Collection-Level readonly (Basis)
HiddenFields []string // Collection-Level hidden (Basis)
Fields []CollectionField
Permissions map[string]PermissionSet
// ...
}
// CollectionField (collection.go)
type CollectionField struct {
Name string
Type string
Readonly *BoolOrEval // Field-Def: true/false (statisch) oder {eval: "..."} (dynamisch)
Hidden *BoolOrEval // Field-Def: true/false (statisch) oder {eval: "..."} (dynamisch)
SubFields []CollectionField
// ...
}
// BoolOrEval (collection.go) — unterstützt bool oder eval-Ausdruck
type BoolOrEval struct {
Value *bool // statischer Wert (true/false)
Eval *string // JS-Ausdruck als String
JsProgram *JsProgram // vorkompiliertes goja-Programm (intern)
}
// FieldDef (utils/field_filter.go) — import-sichere Kopie für die Engine
type FieldDef struct {
Name string
SubFields []FieldDef
Readonly *bool
Hidden *bool
}
```
### Kernfunktionen
```go
// Kaskaden-Auflösung (utils/field_filter.go)
func ResolveEffectiveFields(
collectionLevel []string,
permSetLevel []string,
fields []FieldDef,
fieldDefFn func(f FieldDef) *bool,
) []string
// Prüfung ob geschützte Felder im Body vorhanden sind
func CheckFieldsPresent(doc map[string]interface{}, fields []string) []string
// Prüfung für Operator-Bodies ($set, $inc, etc.)
func CheckOperatorFieldsPresent(doc map[string]interface{}, fields []string) []string
// Felder aus Response-Dokument entfernen (für GET)
func StripFields(doc map[string]interface{}, fields []string)
// Feld-Definitionen aus Config entfernen (für enrichData)
func StripFieldDefsFromConfig(fields []FieldDef, hidden []string) []FieldDef
// Model-Wrapper (models/field_permissions.go)
func ResolveEffectiveReadonlyFields(collConfig, permSet, fields) []string
func ResolveEffectiveHiddenFields(collConfig, permSet, fields) []string
func CollectionFieldsToFieldDefs(fields []CollectionField) []FieldDef
func StripFieldsFromConfig(fields []CollectionField, hidden []string) []CollectionField
// Eval-Funktionen (models/field_permissions.go)
func EvalFieldReadonly(doc, collConfig, dbAPI, auth, method) ([]string, error)
func EvalFieldHidden(doc, collConfig, dbAPI, auth, method) ([]string, error)
func HasEvalFields(fields []CollectionField) bool
func HasEvalReadonlyFields(fields []CollectionField) bool
func HasEvalHiddenFields(fields []CollectionField) bool
// Vorkompilierung (models/api.go)
func compileFieldEvals(colName, fields, moduleCache, errs)
```
---
## Abwärtskompatibilität
- **Vollständig abwärtskompatibel**: Configs ohne `readonlyFields`/`hiddenFields` verhalten sich identisch
- `GetEffectivePermissionSet()` gibt für nicht vorkonfigurierte Felder leere Listen zurück → keine Checks
- `validProjections` arbeitet orthogonal und bleibt unberührt
- Admin-Bypass ist konsistent mit dem bestehenden Permission-System
---
## Tests
### Unit-Tests
| Datei | Tests | Abdeckung |
|-------|-------|-----------|
| `internal/utils/field_filter_test.go` | 22 | `ResolveEffectiveFields`, `CheckFieldsPresent`, `StripFields`, `StripFieldDefsFromConfig`, `CheckOperatorFieldsPresent` |
| `internal/models/field_permissions_test.go` | 22 | `CollectionFieldsToFieldDefs`, `ResolveEffectiveReadonlyFields`, `ResolveEffectiveHiddenFields`, `StripFieldsFromConfig` |
### Integration-Tests
| Datei | Tests | Abdeckung |
|-------|-------|-----------|
| `internal/controller/collection_field_permission_test.go` | 31 | Alle Endpoints, alle Rollen (statische Permissions) |
| `internal/controller/collection_field_eval_test.go` | 16 | Eval-basierte Permissions (readonly + hidden) |
**Test-Kategorien (statisch):**
| Kategorie | Tests | Was wird geprüft |
|-----------|-------|-----------------|
| POST readonly | 6 | User/Collection-Level/Field-Def/NeverReadonly/Admin-Bypass/Erlaubte Felder |
| POST hidden | 4 | User/Collection-Level/Field-Def/Admin-Bypass |
| PUT readonly | 3 | User-Rejection/Erlaubte Felder/Admin-Bypass |
| PUT Operator | 3 | $set readonly/$inc hidden/Erlaubte $inc |
| PUT hidden | 1 | User hidden field rejection |
| GET hidden | 5 | User-Strip/Admin-Vollzugriff/Public-Strip/SingleDoc/Readonly-sichtbar |
| Token | 4 | Readonly-Rejection/Erlaubte Felder/Hidden-Rejection/GET-Strip |
| yourPermissions | 2 | User (readonlyFields ja, hiddenFields nein) / Admin (hiddenFields-Hint) |
| Config-Delivery | 2 | User (hidden Feld-Defs entfernt) / Admin (alle sichtbar) |
| Mutation-Safety | 1 | Shared API nach User-Request nicht mutiert |
**Test-Config:** `testdata/api/testproject_fieldperm/collections/orders.yml`
**Test-Kategorien (Eval):**
| Kategorie | Tests | Was wird geprüft |
|-----------|-------|-----------------|
| POST eval readonly | 4 | Bedingung false=erlaubt, true=rejected, admin-bypass, ohne Feld=erlaubt |
| POST eval hidden | 2 | Bedingung false=erlaubt, true=rejected |
| POST statisch | 2 | Statisches readonly/hidden weiterhin korrekt |
| PUT eval readonly | 3 | Bedingung false=erlaubt, true=rejected, admin-bypass |
| GET eval hidden | 5 | Bedingung false=sichtbar, true=stripped, admin-sieht-alles, mixed-results, statisch-immer-stripped |
**Test-Config:** `testdata/api/testproject_fieldperm/collections/evalorders.yml`
src/content/docs/server/18-orgs-teams.md
+720
View File
@@ -0,0 +1,720 @@
---
title: 'Orgs & Teams'
---
# Orgs & Teams
## Overview
Tibi-Server uses a 3-level hierarchy for access control and data isolation: **Org → Team → User**. Orgs define the visibility boundary (which projects a user can see), while teams define working rights (what a user can do within a project's collections).
```text
┌─────────────────────────────────────────────────┐
│ Org │
│ (Visibility Boundary / Data Isolation) │
│ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Team Alpha │ │ Team Beta │ │
│ │ (Working │ │ (Working │ │
│ │ Rights) │ │ Rights) │ │
│ │ │ │ │ │
│ │ 👤 User A │ │ 👤 User B │ │
│ │ 👤 User C │ │ 👤 User C │ ← multi │
│ └──────┬───────┘ └──────┬───────┘ team │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────┐ │
│ │ Project X │ │
│ │ teams: [Alpha, Beta] │ │
│ │ orgId: this-org │ │
│ └──────────────────────────────┘ │
└─────────────────────────────────────────────────┘
```
### Key Concepts
| Concept | Description |
|---------|-------------|
| **Org** | Isolation boundary. Org members can see all projects of their org. |
| **Team** | Working unit. Teams are assigned to projects and carry permission sets. |
| **OrgMembership** | Links a user to an org with a role (`admin` or `member`). |
| **TeamMembership** | Links a user to a team with a role (`admin` or `member`). |
| **Project.orgId** | The owning org of a project. |
| **Project.teams[]** | Which teams have working rights on this project. |
| **User.primaryOrgId** | The user's primary org (used for LLM budget billing). |
### Role Hierarchy
| Role | Scope | Description |
|------|-------|-------------|
| **SystemAdmin** (`role: 0`) | Global | Full access to everything. Bypasses all access control. |
| **OrgAdmin** (`orgs[].role: "admin"`) | Per Org | Can manage teams and users within their org(s). Can assign teams to projects. |
| **OrgMember** (`orgs[].role: "member"`) | Per Org | Can see projects of their org. Working rights come from team membership. |
| **TeamAdmin** (`teams[].role: "admin"`) | Per Team | Currently same as TeamMember. Reserved for future use. |
| **TeamMember** (`teams[].role: "member"`) | Per Team | Gets collection-level working rights when team is assigned to a project. |
## Data Model
### Org
```json
{
"id": "507f1f77bcf86cd799439011",
"name": "Acme Corp",
"llmBudgets": [
{
"provider": "openai",
"tokenBudget": 500000,
"tokensUsed": 123456,
"periodStart": "2026-01-01T00:00:00Z",
"periodEnd": "2026-04-01T00:00:00Z"
}
],
"meta": { "industry": "Tech" },
"insertTime": "2026-01-15T10:00:00Z",
"updateTime": "2026-01-15T10:00:00Z"
}
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | yes | Unique org name. Supports text search. |
| `llmBudgets` | LLMBudget[] | no | Org-level LLM token budgets (see [LLM Budget Cascade](#llm-budget-cascade)). |
| `meta` | object | no | Arbitrary metadata. |
**Collection name:** `org`
### Team
```json
{
"id": "507f1f77bcf86cd799439012",
"name": "Editors",
"orgId": "507f1f77bcf86cd799439011",
"permissions": ["editor", "reviewer"],
"meta": { "description": "Content editors" },
"insertTime": "2026-01-15T10:00:00Z",
"updateTime": "2026-01-15T10:00:00Z"
}
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `name` | string | yes | Team name. Must be unique within the org. |
| `orgId` | string | yes | Reference to the parent org. Immutable after creation. |
| `permissions` | string[] | no | Named permission sets that this team grants (must match keys defined in collection configs). |
| `meta` | object | no | Arbitrary metadata. |
**Collection name:** `team`
**Composite identity:** `{name, orgId}` — team names are unique per org, not globally.
### User (extended fields)
The following fields have been added to the User model:
```json
{
"username": "alice",
"role": 1,
"orgs": [
{ "orgId": "507f1f77bcf86cd799439011", "role": "admin" },
{ "orgId": "507f1f77bcf86cd799439013", "role": "member" }
],
"teams": [
{ "teamId": "507f1f77bcf86cd799439012", "role": "member" },
{ "teamId": "507f1f77bcf86cd799439014", "role": "admin" }
],
"primaryOrgId": "507f1f77bcf86cd799439011"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `orgs` | OrgMembership[] | Orgs the user belongs to, each with a role. |
| `teams` | TeamMembership[] | Teams the user belongs to, each with a role. |
| `primaryOrgId` | string | Primary org for LLM budget billing. Must reference an org in `orgs[]`. |
#### OrgMembership
| Field | Type | Values | Description |
|-------|------|--------|-------------|
| `orgId` | string | ObjectID | Reference to the org. |
| `role` | string | `"admin"`, `"member"` | User's role within this org. |
#### TeamMembership
| Field | Type | Values | Description |
|-------|------|--------|-------------|
| `teamId` | string | ObjectID | Reference to the team. |
| `role` | string | `"admin"`, `"member"` | User's role within this team. |
### Project (extended fields)
```json
{
"name": "website",
"namespace": "website_prod",
"orgId": "507f1f77bcf86cd799439011",
"teams": ["507f1f77bcf86cd799439012", "507f1f77bcf86cd799439014"]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `orgId` | string | The owning org. Determines which users can see this project. |
| `teams` | string[] | Team IDs assigned to this project. Members of these teams get working rights. |
### JWT Token Claims (extended)
The JWT access token now includes org and team membership:
```json
{
"object": {
"id": "60a1b2c3d4...",
"username": "alice",
"role": 1,
"permissions": [],
"orgs": [
{ "orgId": "507f1f77bcf86cd799439011", "role": "admin" }
],
"teams": [
{ "teamId": "507f1f77bcf86cd799439012", "role": "member" }
],
"primaryOrgId": "507f1f77bcf86cd799439011"
},
"type": "User",
"exp": 1700000000
}
```
## API Endpoints
### Org CRUD
All org endpoints require authentication via `X-Auth-Token`.
#### List Orgs
```text
GET /api/v1/org
```
**Visibility:**
- SystemAdmin: sees all orgs
- Others: sees only orgs they are a member of
**Query parameters:** Standard collection query params (`_search`, `filter`, `sort`, `offset`, `limit`).
#### Create Org
```text
POST /api/v1/org
```
**Auth:** SystemAdmin only.
**Request:**
```json
{
"name": "Acme Corp",
"meta": { "industry": "Tech" }
}
```
#### Update Org
```text
PUT /api/v1/org/:id
```
**Auth:** SystemAdmin only.
#### Delete Org
```text
DELETE /api/v1/org/:id
```
**Auth:** SystemAdmin only.
**Cascade effects on delete:**
1. All teams within the org are deleted
2. Team references are removed from all projects (`teams[]`) and users (`teams[]`)
3. Projects are decoupled: `orgId` unset, `teams` cleared
4. Org membership is removed from all users (`orgs[]`)
---
### Team CRUD
#### List Teams
```text
GET /api/v1/team
```
**Visibility:**
- SystemAdmin: sees all teams
- OrgAdmin: sees all teams in orgs they admin, plus their own teams
- Others: sees only teams they are a member of
#### Create Team
```text
POST /api/v1/team
```
**Auth:** SystemAdmin or OrgAdmin of the team's org.
**Request:**
```json
{
"name": "Editors",
"orgId": "507f1f77bcf86cd799439011",
"permissions": ["editor"]
}
```
**Validation:**
- `orgId` is required and must reference an existing org
- `orgId` is **immutable** after creation
- Team name must be unique within the org
#### Update Team
```text
PUT /api/v1/team/:id
```
**Auth:** SystemAdmin or OrgAdmin of the team's org.
**Note:** `orgId` cannot be changed after creation.
#### Delete Team
```text
DELETE /api/v1/team/:id
```
**Auth:** SystemAdmin or OrgAdmin of the team's org.
**Cascade effects on delete:**
1. Team reference is removed from all projects (`teams[]`)
2. Team membership is removed from all users (`teams[]`)
---
### User Management (org/team-scoped)
#### Assign User to Org
```text
PUT /api/v1/user/:id
```
Add or modify org membership:
```json
{
"orgs": [
{ "orgId": "507f1f77bcf86cd799439011", "role": "admin" },
{ "orgId": "507f1f77bcf86cd799439013", "role": "member" }
]
}
```
**Auth by role:**
| Actor | Can Assign | Restrictions |
|-------|-----------|-------------|
| SystemAdmin | Any org, any role | None |
| OrgAdmin | Orgs they admin | Cannot modify entries for orgs outside their scope. Existing out-of-scope entries must remain unchanged. |
| Others | — | Cannot modify `orgs`. |
#### Assign User to Team
```text
PUT /api/v1/user/:id
```
```json
{
"teams": [
{ "teamId": "507f1f77bcf86cd799439012", "role": "member" }
]
}
```
**Auth:** Same rules as org assignment. OrgAdmin can only assign teams that belong to orgs they admin.
#### Set Primary Org
```text
PUT /api/v1/user/:id
```
```json
{
"primaryOrgId": "507f1f77bcf86cd799439011"
}
```
**Validation:** The referenced org must be in the user's `orgs[]` array. Can be set to `null` to clear.
#### User Visibility
- SystemAdmin: sees all users
- Org members: see all users that share at least one org (`orgs.orgId: {$in: [myOrgIDs]}`)
- No orgs: can only see own profile
---
### Project Team Assignment
```text
PUT /api/v1/project/:id
```
```json
{
"teams": ["507f1f77bcf86cd799439012", "507f1f77bcf86cd799439014"]
}
```
**Auth:**
- SystemAdmin: can set `orgId`, `teams`, and all other fields
- OrgAdmin: can only modify `teams[]`, and only with teams that belong to the project's org
**Validation:** All team IDs in `teams[]` must reference teams that belong to the same org as the project (`project.orgId`).
## Permission System
### How Team Permissions Map to Collections
Teams carry a `permissions[]` array of named permission set keys. These keys correspond to permission sets defined in collection configs:
**Collection config (YAML):**
```yaml
permissions:
public:
methods:
GET: { allow: true }
user:
methods:
GET: { allow: true }
POST: { allow: true }
editor:
methods:
GET: { allow: true }
POST: { allow: true }
PUT: { allow: true }
DELETE: { allow: true }
```
**Team definition:**
```json
{
"name": "Content Editors",
"orgId": "...",
"permissions": ["editor"]
}
```
When a user who is a member of the "Content Editors" team accesses a collection, they get the `editor` permission set — allowing GET, POST, PUT, and DELETE.
### Permission Resolution Priority
When determining access for a collection request, the following priority chain is evaluated top-to-bottom. The first match wins:
```text
1. Admin Token (namespace-level)
└─ Checked via X-Admin-Token header with namespace.*.collection.* permissions
2. Token Permission (query/header)
└─ Checked via ?token= or X-Collection-Token header
3. Public Permission Set
└─ The "public" key in the collection's permissions config
4. SystemAdmin (role=0)
└─ Always allowed — bypasses all checks
5. Team "user" Permission
└─ If user is a member of a team assigned to this project
AND the collection has a "user" permission set → apply it
6. Custom User Permissions
└─ User's auth.permissions[] matched against "namespace:permKey"
7. Custom Team Permissions (NEW)
└─ Team's permissions[] matched against collection permission keys
(e.g. team has ["editor"] and collection has an "editor" set)
8. Denied
```
### Effective Permission Set Selection
For field-level access control and projections, the system selects ONE effective permission set:
| Priority | Condition | Selected Set |
|----------|-----------|-------------|
| 1 | Admin token or SystemAdmin | `nil` (no restrictions) |
| 2 | Token permission matches | Token's permission set |
| 3 | User custom permission matches `namespace:key` | That custom permission set |
| 4 | Team permission key matches collection key | That team permission set |
| 5 | User is team member | `"user"` permission set |
| 6 | Fallback | `"public"` permission set |
### Permission Union from Multiple Teams
When a user is a member of multiple teams assigned to the same project, the permissions are **merged (union)**:
```text
Team Alpha: permissions=["editor"]
Team Beta: permissions=["reviewer"]
User in both → permissions checked: ["editor", "reviewer"]
```
If either team's permission set allows an operation, access is granted.
## Visibility Rules
### Project Visibility
```text
SystemAdmin (role=0) → sees ALL projects
User (role=1) → sees projects where project.orgId ∈ user's org IDs
No org membership → sees NO projects
```
Project visibility is **org-scoped**, not team-scoped. Being a member of an org grants visibility to all projects of that org, regardless of team assignment. Team assignment grants working rights, not visibility.
### Org Visibility
```text
SystemAdmin → sees ALL orgs
Others → sees only orgs they are a member of
```
### Team Visibility
```text
SystemAdmin → sees ALL teams
OrgAdmin → sees all teams in orgs they admin + their own teams
Others → sees only teams they are a member of
```
### User Visibility
```text
SystemAdmin → sees ALL users
Org members → sees users sharing at least one org
No orgs → sees only own profile
```
## Audit Visibility
Audit log access uses the team→project→collection chain (not org-based):
1. System collects all team IDs from the user's JWT
2. Finds all projects where `teams[]` overlaps with the user's teams
3. For each project, checks GET permission per collection
4. Builds a filter of allowed `{projectNamespace, collection}` pairs
5. Non-admin users **never** see `source.type: "system"` entries (login, reload, shutdown events)
```text
SystemAdmin → sees ALL audit entries (including system events)
Team member → sees audit entries for collections they have GET access to
No teams → sees NO audit entries
```
## LLM Budget Cascade
The LLM token budget system uses a two-level cascade: **User → Org**.
### Budget Check Flow
```text
1. Read user.llmBudgets[provider] ← REQUIRED (hard check)
└─ No budget → 429 Too Many Requests
└─ Period expired → 429
└─ Tokens exhausted → 429
2. Read org.llmBudgets[provider] ← OPTIONAL (soft check)
└─ via user.primaryOrgId
└─ No primaryOrgId → skip org check
└─ Org has no budget for provider → no restriction
└─ Org budget expired → 429
└─ Org tokens exhausted → 429
3. Execute LLM request
4. Record usage:
└─ Deduct from user.llmBudgets
└─ Deduct from org.llmBudgets (if applicable)
└─ Write LLMUsageLog with userId AND orgId
```
### Configuration Example
**Org with budget:**
```json
{
"name": "Acme Corp",
"llmBudgets": [{
"provider": "openai",
"tokenBudget": 1000000,
"tokensUsed": 0,
"periodStart": "2026-01-01T00:00:00Z",
"periodEnd": "2026-04-01T00:00:00Z"
}]
}
```
**User with personal budget and primary org:**
```json
{
"username": "alice",
"primaryOrgId": "acme-org-id",
"llmBudgets": [{
"provider": "openai",
"tokenBudget": 50000,
"tokensUsed": 0,
"periodStart": "2026-01-01T00:00:00Z",
"periodEnd": "2026-04-01T00:00:00Z"
}]
}
```
When Alice makes an LLM request:
- Her personal budget is checked first (hard check)
- Then Acme Corp's org budget is checked (soft check via `primaryOrgId`)
- Usage is deducted from both budgets
- The usage log entry includes both `userId` and `orgId`
### Shared Budget Scenario
Multiple users can share an org's budget:
```text
Org "Acme": tokenBudget=1000, tokensUsed=985
Alice (primaryOrgId=Acme): personal budget OK
Bob (primaryOrgId=Acme): personal budget OK
Alice sends request → 15 tokens used → Org tokensUsed becomes 1000
Bob sends request → 429 (org budget exhausted)
Alice sends request → 429 (org budget exhausted)
```
### Team Membership is Irrelevant for Billing
LLM billing goes through `user.primaryOrgId`, not through team membership. A user in multiple orgs/teams is always billed to their primary org:
```text
User in OrgA (primary) + OrgB → billing goes to OrgA only
```
## Cascade Consistency
### Org Deletion Cascade
When an org is deleted, the following cleanup occurs automatically:
1. **Teams**: All teams with `orgId == deletedOrg` are deleted
2. **Projects**: `orgId` is unset, `teams[]` is cleared for all affected projects
3. **Users**: Org membership removed from `orgs[]`, plus:
- Teams belonging to the deleted org are removed from `teams[]`
- `primaryOrgId` is cleared if it referenced the deleted org
### Team Deletion Cascade
When a team is deleted:
1. **Projects**: Team ID removed from `teams[]`
2. **Users**: Team membership removed from `teams[]`
### User Org Removal Consistency
When a user's `orgs[]` is modified and an org is removed:
1. `primaryOrgId` is cleared if it referenced the removed org
2. Teams belonging to the removed org are automatically removed from `teams[]`
## Access Control Summary
### Org Endpoints
| Action | SystemAdmin | OrgAdmin | Member | Unauthenticated |
|--------|-------------|----------|--------|-----------------|
| GET (list) | All | Own orgs | Own orgs | — |
| POST | ✅ | — | — | — |
| PUT | ✅ | — | — | — |
| DELETE | ✅ | — | — | — |
### Team Endpoints
| Action | SystemAdmin | OrgAdmin | Member | Unauthenticated |
|--------|-------------|----------|--------|-----------------|
| GET (list) | All | Org teams + own | Own teams | — |
| POST | ✅ | Own org teams | — | — |
| PUT | ✅ | Own org teams | — | — |
| DELETE | ✅ | Own org teams | — | — |
### User Endpoints (org/team-related)
| Action | SystemAdmin | OrgAdmin | TeamMember |
|--------|-------------|----------|------------|
| GET (list) | All | Shared-org users | Shared-org users |
| POST (create) | ✅ | ✅ (scoped) | — |
| PUT (modify orgs/teams) | ✅ | Own scope only | Own profile only |
| DELETE | ✅ | — | — |
### Project Endpoints
| Action | SystemAdmin | OrgAdmin | Member |
|--------|-------------|----------|--------|
| GET (list) | All | Org projects | Org projects |
| POST | ✅ | — | — |
| PUT (teams) | ✅ | ✅ (own org teams) | — |
| PUT (orgId, name, ...) | ✅ | — | — |
| DELETE | ✅ | — | — |
## Field Protection
When a non-SystemAdmin modifies a user, the following fields are protected:
| Actor | Protected Fields |
|-------|-----------------|
| **OrgAdmin** | `permissions`, `llmBudgets` — cannot modify. `orgs`, `teams`, `primaryOrgId` — scoped to own orgs. |
| **TeamAdmin/Member** | `permissions`, `llmBudgets`, `orgs`, `teams`, `role`, `primaryOrgId` — all protected. Can only modify own profile's basic fields. |
### OrgAdmin Scope Enforcement
When an OrgAdmin modifies a user's `orgs[]` or `teams[]`:
- **In-scope entries** (orgs the admin manages): can be added, removed, or role-changed
- **Out-of-scope entries** (other orgs): must remain unchanged — same number of entries, same roles
- Violation → `403 Forbidden`
Example: OrgAdmin of OrgA edits a user who is in OrgA and OrgB:
```json
// Before:
{ "orgs": [{"orgId": "A", "role": "member"}, {"orgId": "B", "role": "admin"}] }
// Allowed (only OrgA changed):
{ "orgs": [{"orgId": "A", "role": "admin"}, {"orgId": "B", "role": "admin"}] }
// Forbidden (OrgB modified):
{ "orgs": [{"orgId": "A", "role": "member"}, {"orgId": "B", "role": "member"}] }
```
src/content/docs/server/19-actions.md
+315
View File
@@ -0,0 +1,315 @@
---
title: 'Actions'
---
# Actions
## Übersicht
Actions sind Hook-basierte Endpunkte ohne eine zugrunde liegende Datenbank-Collection. Sie eignen sich für Logik, die nicht an ein CRUD-Datenmodell gebunden ist, z.B. Kontaktformulare, Newsletter-Anmeldungen, Webhook-Empfänger oder benutzerdefinierte API-Endpunkte.
Actions verwenden dasselbe Permission-, Hook-, CORS- und Audit-System wie Collections, haben aber **keine** Fields, Indexes, Uploads, Projections, Search oder Bulk-Operationen.
## Konfiguration
Actions werden im Projekt-Config unter dem `actions:`-Block definiert, analog zu `collections:`:
```yaml
# config.yml
namespace: myproject
collections:
- !include collections/events.yml
actions:
- !include actions/contact-form.yml
- !include actions/newsletter-signup.yml
```
### Action-Config
```yaml
# actions/contact-form.yml
name: contact-form
meta:
label: { de: "Kontaktformular", en: "Contact Form" }
description: { de: "Verarbeitet Kontaktanfragen", en: "Processes contact requests" }
permissions:
public:
methods:
post: true
user:
methods:
post: true
get: true
hooks:
post:
bind:
type: javascript
file: hooks/contact-form/post_bind.js
validate:
type: javascript
file: hooks/contact-form/post_validate.js
handle:
type: javascript
file: hooks/contact-form/post_handle.js
return:
type: javascript
file: hooks/contact-form/post_return.js
get:
handle:
type: javascript
file: hooks/contact-form/get_handle.js
cors:
allowOrigins: ["*"]
```
### Felder
| Feld | Typ | Beschreibung |
|------|-----|-------------|
| `name` | string | Eindeutiger Name der Action (URL-Pfad-Segment) |
| `meta` | object | Metadaten wie Label, Description für Admin-UI |
| `permissions` | object | Identisch zu Collection-Permissions (public/user/role-basiert) |
| `hooks` | object | Hook-Definitionen pro HTTP-Methode und Step |
| `cors` | object | Optionale CORS-Konfiguration (überschreibt Projekt-CORS) |
| `audit` | object | Optionale Audit-Konfiguration (siehe Audit-Abschnitt) |
## Admin-UI-Meta (tibi-admin-nova)
Der Server liefert `meta` unverändert als JSON an den Admin aus (Passthrough — keine Validierung, kein Stripping, keine serverseitige Auswertung). Alles unter `meta.methods` ist reine UI-Konvention für Nova; der Hook entscheidet allein über das Verhalten, Nova erkennt es an der Response. Guards: Nova zeigt Sektion und Features nur bei vorhandener Meta (defensiv gegen alte Server ohne Meta).
```yaml
# actions/import-snapshot.yml (Auszug)
meta:
label: { de: "Snapshot importieren", en: "Import Snapshot" }
muiIcon: file-upload-outline
description: { de: "Importiert eine CSV-Datei als Testlauf.", en: "Imports a CSV file as a test run." }
group: system
methods:
post:
data:
fields:
- name: file
type: file
validator:
required: true
accept: [".csv", "text/csv"]
maxFileSize: "2KB"
execution:
kind: sync # nur dokumentativ
output:
mapping:
summary:
fields:
- source: result.imported
type: number
```
Regeln:
- `query.fields` / `data.fields`: Collection-Feldsprache (`type`, `label`, `validator`, `dependsOn` …). Nova rendert über dieselben Widgets und validiert clientseitig (`required`, `accept`, `maxFileSize`, Datei-Hinweise aus `accept`). Serverseitig gilt nichts automatisch — wer serverseitig härten will, schreibt einen `validate`-Hook in JS.
- Datei-Inputs immer inline als Base64/Data-URI im JSON-Body (Nova-Primärpfad, kein Multipart).
- SSE/Progress: Der Hook streamt bei Bedarf auf derselben Route (`context.response.header("Content-Type", "text/event-stream")`, `writeSSE`, `flush`); Nova erkennt `text/event-stream` per Content-Type und zeigt Fortschritt (`output.mapping.progress` mit `statusField`/`messageField`/`indexField`/`countField`, Progress-Text nur aus explizit konfigurierten Status-/Message-Feldern). `execution.kind` (`sync`/`sse`) ist nur dokumentativ, Transport entscheidet der Hook.
- `output.mapping.summary`/`details`: kompakte Ergebnisdarstellung über `source`-Pfade (dot-Notation, `name` nur Legacy-Fallback); ohne Mapping zeigt Nova Raw-JSON.
- Audit-Verlauf: Einträge liegen unter Collection `_actions/<action-name>` und sind in Nova per Deep-Link (`/projects/:id/audit?collection=_actions/<name>`) erreichbar.
## Routing
Actions sind unter folgendem Pfad erreichbar:
```
POST /api/v1/_/:project/_actions/:action
GET /api/v1/_/:project/_actions/:action
```
Der `_actions`-Prefix verhindert Namenskonflikte mit Collections.
## Hook-Steps
### POST
| Step | Beschreibung |
|------|-------------|
| `bind` | Liest und transformiert den Request-Body. Kann Eingabedaten in `context` ablegen. |
| `validate` | Prüft die Eingabedaten. Wirft bei Fehlern eine Exception (HTTP 400). |
| `handle` | Führt die eigentliche Business-Logik aus. Kein DB-Roundtrip — alles passiert im Hook. |
| `return` | Transformiert die Ausgabe vor dem Senden der Response. |
Ausführungsreihenfolge: `bind` → `validate` → `handle` → `return`
### GET
| Step | Beschreibung |
|------|-------------|
| `handle` | Erzeugt die Antwortdaten. |
| `return` | Transformiert die Ausgabe vor dem Senden der Response. |
Ausführungsreihenfolge: `handle` → `return`
## Hook-Context
Action-Hooks haben Zugriff auf denselben Context wie Collection-Hooks, **ohne** die collection-/document-spezifischen Objekte:
**Verfügbar:**
- `context.request()` — Request-Daten (body, query, headers, params)
- `context.data` — geparster JSON-Body (bei POST), identisch zum Collection-Verhalten
- `api.*` — Zugriff auf andere Collections via `api.find()`, `api.create()`, etc.
- `http`, `smtp`, `jwt`, `bcrypt`, `crypto`, `cache`, `ratelimit` — alle Standard-Packages
**Nicht verfügbar:**
- `context.collection` — keine Collection vorhanden
- `context.document` — kein Dokument aus der DB
## Response-Verhalten
Die Response einer Action wird wie folgt bestimmt:
1. Wenn der letzte Hook (z.B. `return` oder `handle`) ein Objekt mit `data`-Key zurückgibt, wird `data` als Response verwendet.
2. Wenn ein Objekt ohne `data`-Key zurückgegeben wird, wird das gesamte Objekt als Response verwendet.
3. Wenn kein Hook eine Response liefert, wird `{"ok": true}` zurückgegeben.
### Beispiel: Response mit `data`-Key
```javascript
// hooks/contact-form/post_handle.js
;(function () {
var body = JSON.parse(context.request().body())
// E-Mail senden via smtp Package
context.smtp.sendMail({
to: "info@example.com",
subject: "Kontaktanfrage von " + body.name,
textBody: body.message
})
return { data: { sent: true, message: "Danke für Ihre Nachricht" } }
})()
```
Response: `{"sent": true, "message": "Danke für Ihre Nachricht"}`
## Permissions
Permissions funktionieren identisch zu Collections:
```yaml
permissions:
public:
methods:
post: true # Jeder kann POST aufrufen
get: false # GET nur für eingeloggte User
user:
methods:
post: true
get: true
admin:
methods:
post: true
get: true
```
Rollen, Filter und `$subst`-Expressions werden genauso unterstützt wie bei Collections.
## CORS
CORS-Konfiguration auf Action-Ebene überschreibt die Projekt-CORS-Konfiguration:
```yaml
cors:
allowOrigins: ["https://www.example.com"]
allowMethods: ["POST"]
allowHeaders: ["Content-Type", "X-Auth-Token"]
```
## Migration von Collection-Hacks
**Vorher** (Collection ohne Fields als Action-Workaround):
```yaml
collections:
- name: contact-form
strictFields: false
fields: []
permissions:
public:
methods:
post: true
hooks:
post:
create:
type: javascript
file: hooks/contact-form/handle.js
```
**Nachher** (Native Action):
```yaml
actions:
- name: contact-form
permissions:
public:
methods:
post: true
hooks:
post:
handle:
type: javascript
file: hooks/contact-form/handle.js
```
Vorteile:
- Keine unnötige DB-Collection
- Kein `fields: []`-Hack
- Eigener Hook-Step `handle` statt `create`
- Klarere Semantik und Trennung
## Audit
Actions unterstützen das gleiche Audit-System wie Collections. Audit-Logs werden unter der Collection `_actions/<action-name>` gespeichert.
### Konfiguration
```yaml
# actions/contact-form.yml
name: contact-form
audit:
enabled: true
# Optional: nur bestimmte Methoden auditieren (Standard: ["post", "get"])
actions:
- post
```
Die Konfiguration ist identisch zu `AuditCollectionConfig` bei Collections:
| Feld | Typ | Default | Beschreibung |
|------|-----|---------|-------------|
| `enabled` | bool | `false` | Audit für diese Action aktivieren |
| `actions` | []string | `["post", "get"]` | Welche HTTP-Methoden auditiert werden |
> **Hinweis:** Audit muss zusätzlich global aktiviert sein (`AUDIT_ENABLED=true` oder `audit.enabled: true` in der Server-Config).
### Audit-Einträge
Audit-Einträge für Actions enthalten:
| Feld | Beschreibung |
|------|-------------|
| `collection` | `_actions/<action-name>` (z.B. `_actions/contact-form`) |
| `action` | HTTP-Methode (`post` oder `get`) |
| `snapshot` | JSON-safe Kopie der Hook-Daten (bei POST) |
| `userId`, `username`, `userRole` | Auth-Informationen (falls vorhanden) |
| `timestamp` | Zeitpunkt der Aktion |
### Audit-Abfrage
Audit-Logs für Actions können über den Standard-Audit-Endpunkt abgefragt werden:
```
GET /api/v1/_/:project/audit?filter={"collection":"_actions/contact-form"}
```
src/content/docs/server/20-referenced-read-safety.md
+75
View File
@@ -0,0 +1,75 @@
---
title: 'Referenced Read Safety'
---
# Referenced Read Safety
## Problem
Features that read from a second collection during a request can silently bypass parts of the target collection's normal GET visibility model if they only check collection-level `get` permission.
Typical examples are:
- lookup-like relation expansion
- reverse aggregates / derived counts and sums
- future facet, join, or enrichment features over referenced collections
The dangerous gap is not only collection access itself, but also the field-level read model of the referenced collection.
That includes at least:
- permission-based pre-filters
- valid projections
- static hidden fields
- eval-based hidden fields
- read and return hooks
- user-supplied target filters that reference unreadable fields
- derived fields such as aggregate `valueField`
## Project Invariant
Any feature that reads or derives data from a referenced collection must go through the shared helper in [internal/controller/collection_reference_access.go](../internal/controller/collection_reference_access.go).
Today that shared path is used by:
- lookup target resolution
- aggregate target resolution
The helper centralizes:
- target collection existence and `get` permission checks
- effective pre-filter resolution
- projection validation / default projection selection for referenced reads
- hidden-field and projection-based field readability checks
- eval-hidden blocking for derived field access
- unreadable target field detection inside aggregate-style filters
- readable target join-field checks for lookup-style `foreignField` access
## Practical Rule
If a new feature touches a referenced collection, do not read target documents directly after `MethodIsAllowed` and `getCollectionPreFilter`.
Instead:
1. resolve referenced read access through the shared helper
2. validate any requested target field against that access model
3. validate any user-controlled target filter against readable fields
4. only then run target read hooks, DB reads, and return hooks
For lookup specifically:
- `foreignField` is a target read surface and must be readable for the caller
- static hidden fields may be pushed down into the referenced projection when that projection can express them safely
- eval-hidden stays a runtime sanitization concern and must not be treated as projection-only safe
## Why This Matters
Without this invariant, a feature can leak information even when the caller never receives the raw target document.
Examples:
- summing a hidden numeric field
- counting only documents where a hidden target field matches a secret condition
- using a target field blocked by projections as an aggregate input
Those are still data leaks, even if the endpoint only returns `_aggregate` values.
src/content/docs/server/21-backup-restore.md
+534
View File
@@ -0,0 +1,534 @@
---
title: 'Backup & Restore'
---
# Backup & Restore
## Overview
Die Backup & Restore-API ermöglicht das Exportieren eines kompletten Projekts (Collections + Media)
als ZIP-Datei sowie den Import auf dem gleichen oder einem anderen Server.
**Nur Admin-User** können diese Endpunkte nutzen (Server-Token, Projekt-Token mit `allowBackup`,
oder JWT mit AdminRole).
---
## Authentifizierung
Dreistufiges Auth-Modell (identisch mit anderen Admin-Endpunkten wie `reload`):
### 1. Server-Level Admin Token
```yaml
api:
adminTokens:
- token: "backup-agent-token"
label: "ci-backup"
permissions:
- project.backup # Nur Backup/Restore
# - project.* # Alle Projekt-Operationen inkl. Backup
```
Gesendet via `X-Admin-Token` Header oder `Authorization: Bearer <token>`.
### 2. Projekt-Level Admin Token
```yaml
# In der Projekt-Config (config.yml)
admin:
tokens:
- token: "ADMINTOKEN"
allowReload: true
allowRegenerateSearch: true
allowBackup: true # NEU: erlaubt Backup/Restore
```
Das Feld `allowBackup` steuert gezielt, ob dieser Token Backup/Restore verwenden darf.
### 3. JWT Login mit AdminRole
User mit `role: 0` (AdminRole) können Backup/Restore ausführen.
---
## Endpunkte
| Methode | Pfad | Beschreibung |
|--------|------|-------------|
| **GET** | `/_/:project/_/admin/backup` | Backup als ZIP herunterladen |
| **GET** | `/_/:project/_/admin/backup/download/:file` | Temp-ZIP aus SSE-Download abholen |
| **POST** | `/_/:project/_/admin/backup` | ZIP hochladen & analysieren → `restoreToken` (SSE via `?sse=1`) |
| **POST** | `/_/:project/_/admin/backup/:restoreToken` | Wiederherstellung ausführen |
| **DELETE** | `/_/:project/_/admin/backup/:restoreToken` | Temp-Daten manuell aufräumen |
---
## 1. Backup Download
```text
GET /_/:project/_/admin/backup?collections=contacts,pages&sse=1
```
| Query-Param | Default | Beschreibung |
|-------------|---------|-------------|
| `collections` | _(alle)_ | Komma-separierte Collection-Namen (selektiver Export) |
| `sse` | `0` | `1` = Fortschritt als SSE-Stream, `0` = direktes ZIP |
### Direkter Download (`sse=0`)
Response: `Content-Type: application/zip` mit folgender Struktur:
```
backup-<namespace>-<timestamp>.zip
├── backup-info.json
├── collections/
│ ├── <collection>.json # Array aller Dokumente
│ └── ...
└── media/ # Nur Dateien im konfigurierten uploadPath
└── <collection>/
└── <docId>/
└── <fieldName>/
└── <filename>
```
**Wichtig:** Es werden NUR Media-Dateien exportiert, die tatsächlich im konfigurierten
`uploadPath` der Collection existieren. Fehlende Dateien werden übersprungen und geloggt.
Die Datei `backup-info.json` enthält Metadaten, die beim Import zur Schema-Validierung
verwendet werden:
```json
{
"type": "tibi-backup",
"version": 1,
"sourceNamespace": "mein-projekt",
"createdAt": "2026-06-24T12:00:00Z",
"collections": {
"contacts": {
"count": 42,
"fields": ["email", "firstname", "lastname", "phone"]
}
}
}
```
### SSE-Modus (`sse=1`)
Response-Type: `text/event-stream` mit folgenden Events:
```
event: progress
data: {"phase":"init","totalCollections":3}
event: result
data: {"phase":"done","downloadUrl":"/_/:project/_/admin/backup/download/<token>","size":52428800,"expiresAt":"..."}
```
Der Client erhält eine temporäre Download-URL (TTL: 5 Minuten) und ruft die ZIP
anschließend separat ab:
```bash
# Schritt 1: SSE-Download initiieren
curl -H "X-Admin-Token: $TOKEN" \
"http://localhost:8080/api/v1/_/myproject/_/admin/backup?sse=1"
# Schritt 2: ZIP vom temp-Link abholen
curl -H "X-Admin-Token: $TOKEN" \
-o backup.zip \
"http://localhost:8080/api/v1/_/myproject/_/admin/backup/download/<token>"
```
Der Download-Link ist **One-Time-Use** — ein zweiter Request erhält 404.
---
## 2. Backup Upload & Analyse (Phase 1)
```text
POST /_/:project/_/admin/backup
Content-Type: multipart/form-data
```
**Body:** Feld `file` = backup.zip
| Query-Param | Default | Beschreibung |
|-------------|---------|-------------|
| `sse` | `0` | `1` = Fortschritt via SSE streamen |
### Direkter Modus (`sse=0`)
**Response (200):**
```json
{
"restoreToken": "a1b2c3d4e5f6...",
"sourceNamespace": "quelle-projekt",
"createdAt": "2026-06-24T12:00:00Z",
"expiresAt": "2026-06-24T13:00:00Z",
"collections": {
"contacts": {
"documentCount": 42,
"mediaFileCount": 5,
"backupFields": ["email","firstname","lastname","phone"],
"targetFields": ["email","firstname","lastname","phone","department"],
"warnings": [
"target has new fields not in backup: [department]"
],
"exists": true
},
"legacy_old": {
"documentCount": 7,
"mediaFileCount": 0,
"backupFields": ["data"],
"targetFields": null,
"warnings": ["collection \"legacy_old\" does not exist in target project"],
"exists": false
}
}
}
```
**In dieser Phase werden KEINE Daten geschrieben.** Das ZIP wird nur entpackt und analysiert.
Der `restoreToken` referenziert das Temp-Verzeichnis für die nächste Phase.
### SSE-Modus (`sse=1`)
Response-Type: `text/event-stream` mit folgenden Events:
```
event: progress
data: {"phase":"init","restoreToken":"a1b2c3d4e5f6..."}
event: progress
data: {"phase":"save"}
event: progress
data: {"phase":"extract"}
event: progress
data: {"phase":"validate"}
event: progress
data: {"phase":"analyze","totalCollections":2}
event: result
data: {"phase":"done","restoreToken":"a1b2c3d4e5f6...","sourceNamespace":"quelle-projekt","createdAt":"2026-06-24T12:00:00Z","expiresAt":"2026-06-24T13:00:00Z","collections":{...}}
```
### Schema-Kompatibilitäts-Prüfung
Die Prüfung ist **tolerant** — sie warnt, blockiert aber nicht:
| Situation | Verhalten |
|-----------|-----------|
| Backup & Ziel identisch | Kein Warning |
| Backup hat Felder, Ziel nicht | Warning (MongoDB speichert sie trotzdem) |
| Ziel hat Felder, Backup nicht | Warning (Dokumente bekommen null/leer) |
| Collection existiert nicht im Ziel | `exists: false`, wird beim Restore geskippt |
---
## 3. Restore ausführen (Phase 2)
```text
POST /_/:project/_/admin/backup/:restoreToken?collections=contacts&merge=true&sse=1
```
| Query-Param | Default | Beschreibung |
|-------------|---------|-------------|
| `collections` | _(alle)_ | Welche Collections aus dem Backup importieren |
| `merge` | `false` | `true` = Dokumente updaten, Orphans behalten; `false` = exakter 1:1-Abgleich (Orphans löschen) |
| `sse` | `0` | `1` = Fortschritt via SSE streamen |
### Direkter Restore (`sse=0`)
**Response (200):**
```json
{
"restoredCollections": ["contacts"],
"skippedCollections": ["legacy_old"],
"totalDocuments": 42,
"mediaFiles": 5
}
```
### Restore-Modi
**`merge=false` (Default) — Exakter 1:1-Abgleich:**
- Jedes Dokument aus dem Backup wird via atomarem Upsert (`ReplaceOne`) eingefügt oder ersetzt
- Nach dem Import werden alle Dokumente gelöscht, die vor dem Restore existierten, aber **nicht** im Backup enthalten sind (Orphans)
- Ergebnis: Die Collection ist eine exakte 1:1-Kopie des Backups — jedes Feld, jeder BSON-Typ
- Kein Zeitfenster, in dem die Collection leer ist (kein Pre-Delete mehr)
- Bricht der Import ab, sind bereits verarbeitete Docs korrekt, unverarbeitete unangetastet
**`merge=true` — Überschreiben, Orphans behalten:**
- Existierende Dokumente mit gleicher `_id` werden **ersetzt** (atomares Upsert)
- Neue `_id`s werden **angelegt**
- Dokumente im Ziel, die nicht im Backup sind, bleiben **erhalten**
- Geeignet für partielle Updates / Migration einzelner Collections
### SSE-Modus (`sse=1`)
```
event: progress
data: {"phase":"init"}
event: progress
data: {"phase":"collection","collection":"contacts","action":"importing"}
event: result
data: {"restoredCollections":["contacts"],"skippedCollections":[],"totalDocuments":42,"mediaFiles":5,"phase":"done"}
```
---
## 4. Temp-Daten aufräumen
```text
DELETE /_/:project/_/admin/backup/:restoreToken
```
Explizites Cleanup des Temp-Verzeichnisses. Antwort: `{"message":"ok"}`.
**Automatisches Cleanup:** Ein Background-Job (5-Minuten-Takt) löscht Temp-Verzeichnisse,
deren TTL abgelaufen ist:
- **Restore-Temp-Dirs**: 1 Stunde TTL
- **Download-ZIPs**: 5 Minuten TTL
---
## Typischer Workflow
### Szenario A: Projekt von Staging nach Production kopieren
```bash
# 1. Auf Staging: Backup herunterladen
curl -H "X-Admin-Token: $STAGING_TOKEN" \
-o staging-backup.zip \
"https://staging.example.com/api/v1/_/myproject/_/admin/backup"
# 2. Auf Production: Backup hochladen & analysieren
curl -H "X-Admin-Token: $PROD_TOKEN" \
-F "file=@staging-backup.zip" \
"https://prod.example.com/api/v1/_/myproject/_/admin/backup"
# → {"restoreToken":"abc...","collections":{...}}
# 3. Prüfung der Analyse-Warnings, dann Restore ausführen
curl -H "X-Admin-Token: $PROD_TOKEN" \
-X POST \
"https://prod.example.com/api/v1/_/myproject/_/admin/backup/abc..."
# → {"restoredCollections":["contacts","pages"],"totalDocuments":84,"mediaFiles":12}
```
### Szenario B: Nur eine Collection aktualisieren (Selective Restore)
```bash
# 1. Full Backup herunterladen (enthält alle Collections)
curl -H "X-Admin-Token: $TOKEN" -o full-backup.zip "..."
# 2. Upload & Analyse
RESTORE_TOKEN=$(curl -H "X-Admin-Token: $TOKEN" -F "file=@full-backup.zip" "..." | jq -r .restoreToken)
# 3. Nur "contacts" wiederherstellen
curl -H "X-Admin-Token: $TOKEN" -X POST \
"http://.../_/myproject/_/admin/backup/$RESTORE_TOKEN?collections=contacts"
```
### Szenario C: Mit Merge updaten (bestehende Daten erhalten)
```bash
# merge=true: Backup-Docs ersetzen bestehende, neue hinzufügen,
# aber Docs die nur im Ziel existieren bleiben erhalten
curl -H "X-Admin-Token: $TOKEN" -X POST \
"http://.../_/myproject/_/admin/backup/$TOKEN?merge=true"
```
### Szenario D: Mit SSE-Fortschritt (große Projekte)
```bash
# Download mit Fortschrittsanzeige
curl -H "X-Admin-Token: $TOKEN" -N \
"http://.../_/myproject/_/admin/backup?sse=1"
# → event:progress + event:result mit downloadUrl
# Restore mit Fortschrittsanzeige
curl -H "X-Admin-Token: $TOKEN" -N -X POST \
"http://.../_/myproject/_/admin/backup/$TOKEN?sse=1"
```
---
## ZIP-Struktur (Referenz)
```
backup-<namespace>-<timestamp>.zip
├── backup-info.json # Metadaten + Schema-Fingerprint
├── collections/
│ ├── contacts.json # Array aller Dokumente (mit _id)
│ └── pages.json
└── media/ # Nur Dateien im uploadPath
└── contacts/
└── <documentId>/
└── <fieldName>/
└── <filename>
```
Kein `project.json` oder `config.json` — das Projekt muss auf dem Zielsystem bereits existieren.
---
## Media-Erkennung
Beim Export werden **nur** Dateien berücksichtigt, die:
1. Über ein `file`- oder `file[]`-Feld im Dokument referenziert sind
2. **Tatsächlich** im konfigurierten `uploadPath` existieren (`os.Stat` → OK)
3. Innerhalb des `uploadPath` liegen (Path-Traversal-Schutz)
Algorithmus:
```
Für jedes Dokument einer Collection:
Für jedes Feld vom Typ "file" oder "file[]":
Relativer Pfad aus Feldwert extrahieren
Absoluter Pfad = <uploadPath> / <relativer Pfad>
os.Stat() → existiert → in ZIP aufnehmen
Existiert nicht → skip, Log-Warnung
```
---
## Sicherheit
| Maßnahme | Beschreibung |
|----------|-------------|
| **Auth** | Server-Token, Projekt-Token, oder JWT AdminRole erforderlich |
| **Path Traversal** | `filepath.Clean` + `strings.HasPrefix` bei ZIP-Extraktion und Media-Kopie |
| **Temp-Files** | Automatischer Cleanup nach TTL, keine Akkumulation |
| **One-Time Download** | SSE-Download-Links sind nach einmaligem Abruf ungültig |
| **Media außerhalb uploadPath** | Pfade außerhalb des konfigurierten `uploadPath` werden ignoriert |
---
## Fehlerbehandlung
| HTTP-Status | Ursache |
|-------------|---------|
| **400** | Keine Datei hochgeladen, ungültiges ZIP, fehlende `backup-info.json`, falscher Typ/Version |
| **403** | Fehlende Berechtigung (`project.backup` oder `allowBackup`) |
| **404** | `restoreToken` nicht gefunden oder abgelaufen, Projekt nicht gefunden |
| **410** | Download-Link abgelaufen (nur bei `/download/:file`) |
| **500** | Interner Fehler (DB-Lesefehler, ZIP-Erstellung fehlgeschlagen) |
---
## Temp-Datei-Management
```
Upload (Phase 1) → /tmp/tibi-backup/<restoreToken>/ (TTL: 1h)
Restore (Phase 2) → importiert, dann gelöscht
DELETE /:restoreToken → sofort gelöscht
TTL-Ablauf → Background-Cleanup (alle 5 Min)
SSE Download → /tmp/tibi-backup-download/<token>.zip (TTL: 5 Min)
GET /download/:file → ausgeliefert + sofort gelöscht
```
---
## Konfiguration
### Server-Level Token (config.yml)
```yaml
api:
adminTokens:
- token: "backup-token"
label: "backup-agent"
permissions:
- project.backup
```
`project.*` (Wildcard) inkludiert automatisch `project.backup`.
### Projekt-Level Token
```yaml
admin:
tokens:
- token: "PROJEKT-TOKEN"
allowBackup: true
```
### Keine Projekt-Config-Änderung nötig
Der Backup/Restore-Vorgang benötigt keine Änderungen an der Projekt-Config.
Collections, Felder, uploadPath etc. werden aus der bestehenden Konfiguration gelesen.
---
## Implementierung
### Dateien
| Datei | Inhalt |
|-------|--------|
| `internal/controller/admin_backup.go` | Alle Handler (Download, Analyze, Restore, Cleanup, SSE, Temp-Stores) |
| `internal/controller/admin_backup_auth_test.go` | 12 Auth-Tests |
| `internal/controller/admin_backup_analyze_test.go` | 8 Analyze- & Download-Tests |
| `internal/controller/admin_backup_restore_test.go` | 9 Restore-Tests |
| `internal/controller/admin_backup_integration_test.go` | 3 SSE/E2E-Tests |
| `internal/controller/setup.go` | Routen-Registrierung |
| `internal/models/project.go` | `AdminToken.AllowBackup` Feld |
| `cmd/server/main.go` | `StartBackupCleanupJob()` Aufruf |
### Test-Statistik
**33 Tests** (alle grün):
- 12 Auth-Tests (Server-Token, Projekt-Token, JWT, Berechtigungen)
- 6 Analyze-Tests (ZIP-Validierung, Schema-Diffs, unbekannte Collections)
- 2 Download-Tests (direkt, selektiv)
- 9 Restore-Tests (Import, Merge, Selective, Media-Copy, Cleanup)
- 3 SSE/E2E-Tests (SSE-Download, DownloadFile, SSE-Restore)
- 1 E2E-Test (skipped — Mock-Projekt-API-Limitierung)
Keine Regression auf bestehenden Admin-Tests (`TestAdminReload`, `TestAdminAuthReload` etc.).
---
## Betrieb: Speicherverhalten seit Streaming-Umbau (2026-10)
Erstellung und Wiederherstellung von Snapshots und Backups laufen in
**O(Batch)** statt O(Collection): Keyset-Pagination (`_id $gt`, Sort `_id`),
Datei-/ZIP-Streaming, sequenziell pro Collection. RAM ist keine Grenze mehr —
auch 10-GB-Collections (z. B. Outdoorworks: `productVariant` 91k × ~117 KB mit
`_search`-Bloat) laufen mit konstantem Heap.
### Batch-Konstanten (`internal/controller/snapshot_stream.go`)
| Konstante | Wert | Wirkung |
|---|---|---|
| `SnapshotBatchDocs` | 500 | Docs pro DB-Batch und Restore-Batch |
| `SnapshotBatchBytes` | 8 MiB | reserviert (Byte-Cap, greift pro großem Einzeldoc) |
| `SnapshotBufSize` | 64 KiB | `bufio`-Puffer für Snapshot-Dateien |
| `OrphanMapThreshold` | 200.000 | darunter In-RAM-Orphan-Set, darüber Temp-Collection in Ziel-DB |
| `orphanChunkSize` | 500 | `$in`-Chunks für Orphan-Deletes |
### Empfehlungen
- `GOMEMLIMIT` setzen (z. B. `GOMEMLIMIT=512MiB` im Container): begrenzt GC-Druck,
Streaming braucht es nicht — aber es schützt vor Nachbar-Last.
- Jeder Snapshot-Run loggt `snapshot <id>: done documents=… heapGrowthMB=…`
(`heapGrowthMB` = Heap-Wachstum über den Lauf, kein Peak; Peak ist per
Konstruktion batch-begrenzt).
- Retention `maxCount` behält die **neuesten** Einträge (Bugfix 2026-10-15:
zuvor wurden fälschlich die neuesten gelöscht).
### Nachweise (Tests)
- `TestExportStreaming_PeakScalesFlat` / `TestStreamRestore_PeakScalesFlat`:
10× Docs ⇒ kein 10× Peak (Kontrollmessung Alt-Pfad liegt ein Mehrfaches darüber).
- E2E Roundtrips (Mock + Real-Mongo): Snapshot NonMerge/Merge, Backup
Restore-Matrix, Download-Pipelines.
- OOM-Fixture: `testdata/outdoorworks-oom/` (echte `_search`-Form, Stats,
`dump-full.sh` für den 8-GB-Voll-Dump nach `/WM_Dev/.cortex/mcp-tmp/`).
src/content/docs/server/22-personal-access-tokens.md
+295
View File
@@ -0,0 +1,295 @@
---
title: '22. Personal Access Tokens'
---
# 22. Personal Access Tokens
Self-managed API tokens for users. Every user can create, list, update and delete
their own tokens. A token can be restricted to a subset of the owner's rights and
is **always** checked live against the current rights of its owner.
- [Token format & storage](#token-format--storage)
- [Quick start](#quick-start)
- [Scope model](#scope-model)
- [Scope reference](#scope-reference)
- [Which scopes a user may grant](#which-scopes-a-user-may-grant)
- [Endpoint → scope mapping](#endpoint--scope-mapping)
- [Management API](#management-api)
- [Security properties](#security-properties)
- [Status codes](#status-codes)
Personal Access Tokens are a core feature and always enabled.
## Token format & storage
```text
tibi_pat_<24-hex objectId>_<base64url secret>
```
- Only `sha256(secret)` is stored (`tokenHash`); the plaintext exists exactly once,
in the create response.
- `tokenPrefix` (`tibi_pat_` + 4 secret chars, 13 chars total) is stored for
display and audit.
- Lookup uses the embedded object id → one indexed `_id` read.
- The secret is compared in constant time.
- No expiry by default; `expiresAt` is optional (`expiresAt` must be in the
future when minting).
A PAT is sent via **`X-Auth-Token`** (like the JWT). `Authorization: Bearer` is
reserved for project-specific authentication and is *not* accepted for PATs.
## Quick start
```bash
# 1) create (requires a user login session, not a PAT)
curl -X POST https://host/api/v1/token \
-H "X-Auth-Token: <jwt>" \
-H "Content-Type: application/json" \
-d '{"name":"CI Deploy","scopes":["namespace.website.articles.read"]}'
# 2) use the returned plaintext token (shown exactly once)
curl https://host/api/v1/_/website/articles \
-H "X-Auth-Token: tibi_pat_..."
# 3) revoke (immediately effective)
curl -X DELETE https://host/api/v1/token/<id> -H "X-Auth-Token: <jwt>"
```
## Scope model
A scope is a dotted permission string. Three rules define the effective access of
a PAT request:
1. **Token ⊆ owner.** A scope can only be granted if the owner currently holds
that permission. The catalog is computed live from the database at
create/update time.
2. **Scope ∩ owner (AND).** At request time the action must be allowed by the
**token scope** *and* by the **owner's current rights**. Revoking a user right
(role, org, team, `permissions[]`) restricts the token immediately.
3. **Same syntax as admin tokens.** Dotted segments, `*` wildcard per segment,
and shorthands where fewer segments imply all descendants. Because of rule 2,
even a very broad scope (up to `*`) can never grant more than the owner
currently holds.
**Separate credentials authorize independently.** A valid **server/project admin
token** (`X-Admin-Token` or `Authorization: Bearer`) or a **collection config
token** (`name:value` header/query) authorizes on its own; passing an additional
narrow PAT does not restrict such a request. PAT scopes only restrict requests
that are authenticated *as a user* by the PAT.
Additional semantics:
- **Public baseline.** Data that is publicly readable stays readable with any PAT
(even with no scopes) — a token never adds or removes anonymous access.
- **Empty scope list** is allowed and yields a token that can only reach public
data. `*` yields the full (but still owner-capped) access.
- **Shorthands** are prefixes and never grant more than the owner has.
### Shorthand resolution
| Scope written | Matches (at most the owner's rights) |
|---------------|--------------------------------------|
| `*` | everything the owner may do |
| `namespace` | all collections in all namespaces |
| `namespace.<ns>` | every collection/action in `<ns>` the owner may use |
| `namespace.<ns>.<col>` | every action on `<col>` the owner may use |
| `namespace.<ns>.<col>.<action>` | exactly that action |
| `user` / `project` / `team` / `org` | all actions of that system domain |
| `user.read` / `project.reload` / `audit.read` / `llm.chat` | exactly that action |
| `namespace.*.<col>.read` / `user.*` | wildcard matches any single segment |
Because of rule 2, a shorthand or wildcard can never exceed the owner's rights —
it only avoids enumerating them. `*` is the convenient "everything this user may
do" scope; a request is still rejected as soon as the owner lacks the right.
## Scope reference
### Namespace (project collection) scopes
Namespace = the project's `namespace` (see project config).
| Scope | Grants |
|-------|--------|
| `namespace.<ns>.<col>.read` | `GET /_/<ns>/<col>` (list/one), `_count`, `_facet`, `_queue`, file reads, lookup/aggregate reads of `<col>` (as target), audit entries of `<col>` |
| `namespace.<ns>.<col>.create` | `POST /_/<ns>/<col>` (single and bulk) |
| `namespace.<ns>.<col>.update` | `PUT /_/<ns>/<col>/:id` (single and bulk) |
| `namespace.<ns>.<col>.delete` | `DELETE /_/<ns>/<col>/:id` (single and bulk; bulk additionally needs the owner's bulk right) |
| `namespace.<ns>.<col>` | all of the above the owner holds |
| `namespace.<ns>` | all of the above for every collection in `<ns>` |
| `namespace` | all of the above for every collection the owner may use |
| `*` | everything below and the system domains the owner may use |
### Action endpoint scopes
Actions (`/_/<ns>/_actions/<name>`) have their own scopes.
| Scope | Grants |
|-------|--------|
| `namespace.<ns>._actions.<name>.read` | `GET /_/<ns>/_actions/<name>` |
| `namespace.<ns>._actions.<name>.create` | `POST /_/<ns>/_actions/<name>` |
### System domain scopes
| Scope | Grants |
|-------|--------|
| `user` | all user actions below |
| `user.read` | `GET /api/v1/user`, `GET /api/v1/user/:id` (owner-visibility applied) |
| `user.create` | `POST /api/v1/user` |
| `user.update` | `PUT /api/v1/user/:id` |
| `user.delete` | `DELETE /api/v1/user/:id` |
| `project.read` | `GET /api/v1/project`, `GET /api/v1/project/:id` |
| `project.create` | `POST /api/v1/project` |
| `project.update` | `PUT /api/v1/project/:id` |
| `project.delete` | `DELETE /api/v1/project/:id` |
| `project.reload` | `POST /_/<ns>/_/admin/reload` |
| `project.backup` | `POST/GET/DELETE /_/<ns>/_/admin/backup…` |
| `project.regenerate-search` | `POST /_/<ns>/_/admin/regenerate-search` |
| `project.snapshot` | `/_/<ns>/_/admin/snapshots…` (list/detail/restore/download/prune) |
| `team` / `org` | all actions below |
| `team.read` / `team.create` / `team.update` / `team.delete` | `/api/v1/team…` |
| `org.read` / `org.create` / `org.update` / `org.delete` | `/api/v1/org…` |
| `audit.read` | `GET /api/v1/audit` (see below) |
| `llm.chat` | `POST /api/v1/llm/chat` |
**Audit is special:** a PAT needs `audit.read` **and** the read scope of a
collection (`namespace.<ns>.<col>.read`) to see that collection's audit entries.
`audit.read` alone does not expose collection snapshots. Non-admin users never see
system events (login/reload/shutdown) — this is unchanged by tokens.
**Not scope-relevant (public by design):** `GET /_/<ns>/_/assets/…`, `_meta`,
`_openapi/:project`, `/version`, `GET /api/v1/llm/providers`.
## Which scopes a user may grant
`GET /api/v1/token/_scopes` returns the exact catalog for the calling user. In
general:
| Owner role | System scopes available |
|------------|-------------------------|
| SystemAdmin (`role: 0`) | all of the above, including `project.reload/backup/regenerate-search/snapshot` |
| OrgAdmin (`orgs[].role: "admin"`) | `user.read/create/update`, `team.read/create/update/delete`, `project.read/update`, `org.read`, `audit.read`, `llm.chat` |
| User (`role: 1`) | `user.read/update`, `project.read`, `team.read`, `org.read`, `audit.read`, `llm.chat` |
Collection and action scopes are additionally limited to the projects/collections
the owner can reach (org visibility + team permissions). Public-only rights are
never part of the catalog.
## Endpoint → scope mapping
| Endpoint | Required scope |
|----------|----------------|
| `GET /_/<ns>/<col>` (+ `_count`, `_facet`, `_queue`) | `namespace.<ns>.<col>.read` |
| `POST /_/<ns>/<col>` | `namespace.<ns>.<col>.create` |
| `PUT /_/<ns>/<col>/:id` | `namespace.<ns>.<col>.update` |
| `DELETE /_/<ns>/<col>/:id` | `namespace.<ns>.<col>.delete` |
| `GET /_/<ns>/<col>/:id/:field/*path` (file) | `namespace.<ns>.<col>.read` (public collections: none) |
| `GET /_/<ns>/<col>?lookup=…:<target>` | `namespace.<ns>.<col>.read` **and** `namespace.<ns>.<target>.read` (public target: none) |
| `GET /_/<ns>/<col>?aggregate=…:<target>` | `namespace.<ns>.<col>.read` **and** `namespace.<ns>.<target>.read` (public target: none) |
| `GET /api/v1/<domain>…?lookup=…:<target>` (system lookup) | `<domain>.read` **and** `<target>.read`; missing target scope → target is silently not embedded |
| `GET/POST /_/<ns>/_actions/<name>` | `namespace.<ns>._actions.<name>.read` / `.create` |
| `/api/v1/user…` | `user.<action>` |
| `/api/v1/project…` | `project.<action>` |
| `/_/<ns>/_/admin/reload` etc. | `project.reload` / `.backup` / `.regenerate-search` / `.snapshot` |
| `/api/v1/team…`, `/api/v1/org…` | `team.<action>`, `org.<action>` |
| `GET /api/v1/audit` | `audit.read` (+ per-collection read scope) |
| `POST /api/v1/llm/chat` | `llm.chat` |
`GET /api/v1/token*` itself is **not** scope-controllable: token management
requires a user login session (a PAT gets `403`).
## Management API
All endpoints operate on the caller's own tokens; system admins can see and
manage all.
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/v1/token` | list own tokens (never contains secret/hash) |
| `POST` | `/api/v1/token` | create → returns the plaintext token **once** |
| `GET` | `/api/v1/token/_scopes` | available scopes for the current user |
| `PUT` | `/api/v1/token/:id` | update name/scopes/expiry (re-validated against the owner catalog) |
| `DELETE` | `/api/v1/token/:id` | delete (revoke); effective immediately |
Create response (the `token` field appears only here):
```json
{
"id": "60a1b2c3d4e5f6a7b8c9d0e1",
"name": "CI Deploy",
"tokenPrefix": "tibi_pat_ab12",
"scopes": ["namespace.website.articles.read"],
"expiresAt": null,
"token": "tibi_pat_60a1b2c3d4e5f6a7b8c9d0e1_..."
}
```
## Security properties
- **Never more than the owner:** scopes are validated against the owner's live
rights when minted; every request re-checks the owner's current rights
(scope ∩ owner).
- **Immediate revocation:** deleting a token invalidates it on the next request
(no validity cache); user permission/role/org/team changes take effect
immediately.
- **No secret leakage:** only the hash is persisted; `tokenHash` is never
serialized; audit entries record `authMethod="access-token"`, `tokenLabel`
(name) and `tokenPrefix`, never the secret or hash.
- **Ownership:** `PUT`/`DELETE` are restricted to the owner (or system admin).
- **Session-only management:** `/api/v1/token*` (including `/_scopes`) requires a
user login session; a PAT cannot mint/manage tokens or impersonate users.
- **Impersonation:** tokens cannot be created or managed while impersonating.
- **User deletion:** all tokens of the user are removed (cascade).
## UI integration notes (tibi-admin-nova)
- **Session required:** management calls use the normal login session
(`X-Auth-Token: <jwt>`). PATs are rejected on `/api/v1/token*` with `403`.
- **Scope picker:** `GET /api/v1/token/_scopes` returns `string[]` — the exact
scopes the current user may grant (concrete collection/action scopes, namespace
and collection shorthands, system actions). Offer them as checkboxes plus a
"select all" (= everything the user may do).
- **Power input:** arbitrary admin-token-style scopes are allowed (`*`, `user`,
`namespace.website`). The server validates them and answers `400` for invalid
syntax or scopes unrelated to the user's rights.
- **Create:** `POST /api/v1/token` with `{"name": "...", "scopes": [...],
"expiresAt": null}`. The response contains the plaintext `token` **exactly
once** — show a copy-once dialog and never persist it in the frontend state
longer than needed.
- **List:** `GET /api/v1/token` →
`[{id, name, tokenPrefix, scopes, expiresAt, lastUsedAt, insertTime, updateTime}]`.
Never contains `token` or `tokenHash`.
- **Update:** `PUT /api/v1/token/:id` with any of `name`, `scopes`, `expiresAt`
(re-validated; `expiresAt` must be in the future).
- **Revoke:** `DELETE /api/v1/token/:id` — effective immediately.
- **Errors:** `403` (no/insufficient session, foreign token, missing scope),
`400` (invalid syntax, scope not permitted, past `expiresAt`), `404` (unknown
token id). Show the `error` string.
- **Display list entries with** `name`, `tokenPrefix` and `lastUsedAt`; the
plaintext is never shown again.
## Known limitations
These are accepted, documented residual topics (deliberately not implemented):
- **No rate limit / cap on token creation.** Rate limiting (`ratelimit`) applies
to `POST /api/v1/login` only. An authenticated session can mint an arbitrary
number of tokens; every create/delete is audited, but there is no per-user cap.
If needed, add a limiter to `/api/v1/token` and/or a max-tokens-per-user config.
- **Invalid-token requests cost 1–2 indexed DB reads** (token + owner). Repeated
requests with arbitrary `tibi_pat_…` values are not throttled, which is a mild
amplification vector. Mitigation would be auth-failure rate limiting.
Neither is a privilege escalation: a token never exceeds its owner and never
exposes secrets.
## Status codes
| Situation | Status |
|-----------|--------|
| Missing/invalid/expired/deleted token | `403` |
| Token valid, scope missing | `403` |
| Scope present, owner right missing | `403` |
| `POST`/`PUT` with a syntactically invalid scope (bad segments, unknown domain) | `400` |
| `POST`/`PUT` with a scope not related to any owner right | `400` |
| `expiresAt` in the past | `400` |
| Foreign token `PUT`/`DELETE` | `403` (owner or system admin only) |
src/content/docs/server/23-mcp.md
+156
View File
@@ -0,0 +1,156 @@
---
title: '23. MCP-Server'
---
# 23. MCP-Server
Tibi-server ist zusätzlich ein MCP-Server (Streamable HTTP, JSON-RPC 2.0) für
Agenten, die Websites befüllen. Alle Tools laufen mit denselben Checks wie REST.
- [Scope-Modell](22-personal-access-tokens.md) · [Auth](05-authentication.md) ·
[Collections](04-collections.md) · [Actions](19-actions.md)
## Transport & Auth
```
POST /api/v1/mcp # JSON-RPC: initialize, tools/list, tools/call
GET /api/v1/mcp # Capability-Probe (PAT nötig)
```
- Auth **nur** per PAT: `X-Auth-Token: tibi_pat_...`
- JWT, `X-Admin-Token`, `Authorization: Bearer` (non-PAT) → `403`.
- Deaktiviert (`mcp.enabled: false`) → `404`.
```bash
# Tools listen
curl -X POST https://host/api/v1/mcp \
-H "X-Auth-Token: tibi_pat_..." -H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
# Tool aufrufen
curl -X POST https://host/api/v1/mcp \
-H "X-Auth-Token: tibi_pat_..." -H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/call",
"params":{"name":"collections_list","arguments":{"namespace":"website"}}}}'
```
JSON-RPC-Fehlercodes: `-32601` unbekanntes Tool, `-32004` Resource nicht gefunden,
`-32003` Scope/Permission (`403`), `-32602` Validierung (`400`), `-32603` intern (`500`),
`-32700` Parse-Error.
## Konfiguration (`config.yml`)
```yaml
mcp:
enabled: true
route: "/api/v1/mcp"
maxResultItems: 100
allowAdminActions: true # reload/backup/snapshot nur wenn Token-Scope es erlaubt
```
Env: `MCP_ENABLED`, `MCP_ROUTE`, `MCP_MAX_RESULT_ITEMS`, `MCP_ALLOW_ADMIN_ACTIONS`.
Kein `allowWrite`-Flag (nur PAT-Scopes), kein `llm.chat` via MCP.
## Tools (Hybrid, 36)
Discovery: `projects_list`, `project_get`, `collections_list` (braucht `project.read`),
`collection_describe` (braucht `project.read`; rechtegefilterte Wahrheit: Felder, Validatoren,
`yourPermissions`, Pre-Filter, Projektionen, `meta.agent` ohne Secrets, Perm-Keys maskiert),
`actions_list`, `action_describe`, `jobs_list` (lesend, kein Trigger),
`openapi_get`, `system_meta_get`, `agent_guide_get` (kuratiert aus `meta.agent`, inkl. `yourPermissions` bei Collection),
`assets_list`. `asset_read` ist REST-only (`GET /_/:project/_/assets/...` nutzen).
CRUD: `documents_list` (filter/sort/limit/offset/select/projection/lookup/aggregate/q
plus `extra`-Objekt für `meta.query`-Parameter, s. unten),
`document_get`, `documents_count`, `documents_facet`, `queue_status`,
`document_create` (single + `items[]`-Bulk, Dateien als base64 via `CollectionPost`),
`document_update` (single + `filter`-Bulk), `document_delete`,
`action_call` (nur get/post).
System: `users_list`, `user_get`, `orgs_list`, `org_get`, `teams_list`, `team_get`
(je mit `ValidateFilter`-Sichtbarkeit), `audit_query`
(`audit.read` + Collection-`read`; System-Events für Non-Admin unsichtbar).
REST-only in dieser Phase (klare `-32602`-Meldung, kein Silent-Fail):
`user_create/update/delete`, `project_reload`, `regenerate-search`,
`backup_*`, `snapshot_*`, `file_read`, `asset_read` → REST-Endpunkte nutzen.
Nie via MCP: `token`-Management, `login`/`impersonate`, `shutdown`, `llm.chat`.
## Endpoint → Scope (Auszug)
| MCP-Tool | Scope |
|---|---|
| `projects_list`/`project_get` | `project.read` |
| `collections_list` | `project.read` + je Collection `namespace.<ns>.<col>.read` |
| `collection_describe` | `project.read` + `namespace.<ns>.<col>.read` |
| `documents_*`, `queue_status` | `namespace.<ns>.<col>.read` |
| `document_create` | `namespace.<ns>.<col>.create` |
| `document_update`/`delete` (bulk) | `.update`/`.delete` (+ Owner-Bulk-Recht) |
| `action_call` | `namespace.<ns>._actions.<name>.read`/`.create` |
| `users_*`/`orgs_*`/`teams_*` | `user/org/team.read` (+ Write-Scopes für REST-only) |
| `audit_query` | `audit.read` (+ Collection-`read`) |
Dazu gelten Pre-Filter, Projektionen (`select` nur einengend), Bulk-Flags,
Lookup/Aggregate-Target-Scopes, Validatoren, Hooks (Hooks sind methoden-keyed:
jedes Tool nutzt seine REST-Methode, damit `get`/`put`/`delete`-Hooks laufen)
und Audit-Einträge mit `tokenLabel`/`tokenPrefix` (Client-IP vom Aufrufer) —
identisch zu REST, weil dieselben Controller laufen. MCP-Einträge tragen
zusätzlich `channel: "mcp"` (REST lässt das Feld leer), abfragbar per
`filter={"channel":"mcp"}`.
## Agent-Anleitung (`meta.agent`, YAML-Konvention)
`meta` ist free-form; der Unterschlüssel `agent` ist die kuratierte Anleitung.
Erlaubte Keys: `description`, `purpose`, `instructions`, `frontend`, `workflows`,
`hints`, `fieldHints`, `examples`. Secrets/Token/Pfade/Hooks werden nie
durchgereicht (Größe gecappt).
```yaml
# config.yml
meta:
agent:
description: { de: "Stadtportal", en: "City portal" }
frontend: { routes: "/:slug, /news/:id" }
workflows: ["Artikel anlegen → publizieren → prüfen"]
hints: ["slug im kebab-case"]
# collections/articles.yml
meta:
agent:
purpose: { de: "News-Artikel für Startseite" }
instructions: "Teaser max 160 Zeichen, publishDate immer setzen."
fieldHints: { teaser: "max 160 Zeichen" }
examples: [{ title: "...", published: true }]
```
Idempotenz: `filter`-vor-`create` + `unique`-Indexe; `409` = vorhanden;
stabile Caller-`_id` bei Retry.
## Zusätzliche Query-Parameter (`meta.query`)
Hooks werten oft Query-Parameter aus (`ctx.request().query(name)`). Via MCP sind
nur freigegebene Argumente erlaubt — Collections deklarieren weitere Parameter
in `meta.query` (Schema in `tibi-types/schemas/config/collection-meta.schema.json`):
```yaml
# collections/articles.yml
meta:
query:
hook: { type: string, enum: [fakedata, mail], description: { de: "Test-Dispatcher" } }
view: { type: string, maxLength: 32 }
```
- Typen: `string`, `number`, `boolean`; dazu `enum` und `maxLength`.
- MCP: als `extra`-Objekt an `documents_list`, `document_get`, `documents_count`,
`documents_facet` übergeben — validiert, dann als Query gesetzt. Unbekannte oder
falsch typisierte Keys → `-32602`. Sichtbar in `collection_describe` unter `query`.
- Niemals freigegeben (auch nicht konfigurierbar): Credential-Kanäle
(Permission-Token-Namen, `x-auth-token`, `x-admin-token`, `authorization`) und
Core-Keys (`filter`, `sort`, `limit`, `select`, `lookup`, …).
- REST bleibt unverändert offen; das Schema dient dort als Dokumentation
(`_meta`/OpenAPI) für Agenten. Details: [Collections](04-collections.md#metaquery).
## Beispiele
Siehe `examples/mcp/` (curl: tools/list, describe, Befüllungs-Workflow).
`test.http` enthält MCP-Snippets.
src/content/docs/server/24-ephemeral-rest-tokens.md
+75
View File
@@ -0,0 +1,75 @@
---
title: '24. Ephemere REST-Tokens via MCP (PLAN)'
---
# 24. Ephemere REST-Tokens via MCP (PLAN)
Status: Plan — noch nicht implementiert. Ziel: harness-neutrale Kurzzeit-Credentials,
damit Shell/`curl` alles tun kann, was MCP kann, ohne Base64 in den LLM-Kontext zu ziehen.
## Ausgangslage
- MCP-Uploads (`document_create`/`document_update`, `type:file` als
`{src: "data:<mime>;base64,...", path: "name.jpg"}`) funktionieren, landen aber bei
reinen Tool-Harness (kein JS-Runtime) als ~12k Tokens pro Bild im Prompt.
`documents_list` ohne `select` spült `file.data` ebenso zurück.
- Mit JS-Runtime (OpenCode `execute`: `fetch.bytes()` → `mcp_call`) bleibt Base64 im
Runtime-Speicher, zurück kommt nur `{id, file:{src,path,type}}`. Das hilft nur dort.
- Upload-Tickets (Presigned-PUT, S3, CORS, Quota) wären ein eigenes Projekt.
## Idee
Generische Kurzzeit-Token für alles, was MCP kann, via REST: MCP stellt aus,
Shell verbraucht. Binär/Base64 baut die Shell aus Dateien (`base64`, `jq`, `curl`),
im Prompt stehen nur Token (32 Zeichen) + Pfade.
```bash
# 1) via MCP ausstellen (Scope: ns.collection.method, TTL z. B. 300s)
# -> zurück: tibi_eph_... (einmalig im Klartext)
# 2) in der Shell verbrauchen, ohne LLM-Bloat
curl https://host/api/v1/_/website/articles \
-H "X-Auth-Token: tibi_eph_..." \
-H 'Content-Type: application/json' \
-d @/WM_Dev/.cortex/mcp-tmp/payload.json
curl https://host/api/v1/_/website/medialib/<id>/file/name.jpg \
-H "X-Auth-Token: tibi_eph_..."
```
## Design (angelehnt an 22-PAT)
- Format/Storage wie PAT wiederverwenden (`tokenHash` statt Klartext, `tokenPrefix`
für Anzeige/Audit, Lookup über eingebettete Objekt-ID, Constant-Time-Vergleich),
plus Pflichtfelder: `expiresAt` (max. 1h, Default 5 Min), `ephemeral:true`,
`parentTokenId` (Audit-Verkettung), enger Scope (z. B. genau eine Collection+Methode).
- Scope-Regeln wie PAT übernehmen: **Token ⊆ Owner** bei Ausstellung (live aus DB),
**Scope ∩ Owner** bei jedem Request (Owner-Revoke wirkt sofort). Kein neues
Rechtesystem, keine neuen Header: weiter `X-Auth-Token` wie PAT.
- Transport: neues MCP-Tool `token_issue_ephemeral` (braucht die zu delegierende
Scope-Menge selbst) + optional `token_revoke_ephemeral`. Kein `login`/`impersonate`
via MCP (bleibt verboten wie in 23-MCP). REST-Seite unverändert außer TTL-Check.
- Missbrauchsschutz: kurze TTL + vorhandener `expiresAt`-Pfad, Audit wie PAT,
Rate-Limit auf Ausstellung, kein Refresh (neu ausstellen statt verlängern),
keine Scope-Erweiterung über Owner hinaus.
## Nicht-Ziele
- Kein Presigned-S3, kein Multipart-Spezialfall, keine Änderung an
`handleFileInJSON` (`src`/`path`-Vertrag aus 08-File-Upload bleibt).
- Kein Ersatz für `clear_cache`-Hooks oder `filter_public`-Vertrag.
- Keine langlebigen CI-Token (dafür weiter normale PAT mit `expiresAt`).
## Umsetzungsskizze
1. `token_issue_ephemeral` im MCP (Scope-Check gegen Caller-PAT, TTL clamp 60s–3600s).
2. Token-Dokuments mit `expiresAt` + TTL-Index/Cleanup (prüfen, ob PAT-Cleanup das
schon abdeckt, sonst Job ergänzen).
3. Doku in 22-PAT (Ephemeral-Absatz) + 23-MCP (Tool-Zeile) + Beispiel-Shell-Snippet.
4. Tests: Ablauf nach TTL, Scope-Subset-Verweigerung, Owner-Revoke wirkt sofort,
Audit enthält `parentTokenId`.
## Offene Fragen
- Single-Use vs. Multi-Use innerhalb TTL (Shell-Skripte mögen Multi, Audit mag Single)?
- Soll `project_reload`/`backup`/`snapshot` (heute REST-only, `allowAdminActions`)
auch über Ephemeral erreichbar sein oder explizit ausgenommen bleiben?
- Braucht es IP-/User-Agent-Bindung, oder reicht kurze TTL + enger Scope?
src/content/docs/server/uebersicht.md
+21
View File
@@ -0,0 +1,21 @@
---
title: Tibi-Server Übersicht
description: Was der Tibi-Server kann und wo die Details stehen.
---
Der **Tibi-Server** ist ein dynamisches, konfigurationsgetriebenes Headless-CMS in Go: MongoDB plus YAML-definierte REST-APIs, JavaScript-Hooks, SSE-Realtime, LLM-Chat, Upload/Bilder, PDF, Audit, Cron, OpenAPI, Prometheus, Multi-Tenancy.
## Detailseiten
Die Detailseiten unter `server/` werden per `npm run sync` aus dem Server-Repo (`tibi-server/docs/`) übernommen und hier nur gerendert — Single Source bleibt das Server-Repo:
- Architektur, Konfiguration, API-Endpunkte, Collections
- Authentifizierung, Hooks, Realtime, Upload/Bilder
- LLM-Integration, Audit, Jobs, Deployment
- OpenAPI/Metriken, Security, Troubleshooting, Cookbook
- Feld-Permissions, Orgs/Teams, Aktionen, Backup/Restore, Tokens, MCP
## Konvention für Agenten
- Neue Server-Funktion → erst Code + `tibi-server/docs/*.md`, dann `npm run sync` hier, dann Build prüfen.
- Generierte Referenz (`reference/`) nie von Hand editieren.