huge documentaton restructure for docusaurus

2025-11-20 10:44:29 +01:00
parent 0127114e59
commit 6afa44d947
87 changed files with 18867 additions and 4102 deletions
--- a/docs/ops/README.md
+++ b/docs/ops/README.md
@@ -0,0 +1,13 @@
+# Operations Hub
+
+This section consolidates everything platform operators need: deployment guides, worker management, storage specs, photobooth ingest, and response playbooks. Use it as the entry point when publishing the docs site.
+
+## Structure
+
+- `deployment/` — Docker, Dokploy, and incident playbooks previously under `docs/deployment/`.
+- `photobooth/` — FTP ingest service docs and ops playbooks.
+- `media-storage-spec.md` — Upload/archival flow overview.
+- `guest-notification-ops.md` — Push notification queue monitoring.
+- `queue-workers.md` — Worker container instructions referencing scripts in `/docs/queue-supervisor/`.
+
+Future additions (e.g., escalations, on-call checklists, Terraform notes) should live here as well so all ops content remains in one location.
--- a/docs/ops/deployment/docker.md
+++ b/docs/ops/deployment/docker.md
@@ -0,0 +1,127 @@
+# Docker Deployment Guide
+
+This guide describes the recommended, repeatable way to run the Fotospiel platform in Docker for production or high-fidelity staging environments. It pairs a multi-stage build (PHP-FPM + asset pipeline) with a Compose stack that includes Nginx, worker processes, Redis, and MySQL.
+
+> **Dokploy users:** see `docs/ops/deployment/dokploy.md` for service definitions, secrets, and how to wire the same containers (web, queue, scheduler, vsftpd) inside Dokploy. That document builds on the base Docker instructions below.
+
+## 1. Prerequisites
+
+- Docker Engine 24+ and Docker Compose v2.
+- A `.env` file for the application (see step 4).
+- Optional: an external MySQL/Redis if you do not want to run the bundled containers.
+
+## 2. Build the application image
+
+```bash
+docker compose build app
+```
+
+The build performs the following steps:
+
+1. Installs Node dependencies and runs `npm run build` to produce production assets.
+2. Installs PHP dependencies with Composer (`--no-dev --no-scripts`).
+3. Creates a PHP 8.3 FPM image with required extensions (GD, intl, Redis, etc.).
+4. Stores the compiled application under `/opt/app`; the runtime entrypoint syncs it into the shared volume when a container starts.
+
+## 3. Configure environment
+
+Copy the sample Docker environment file and edit the secrets:
+
+```bash
+cp docker/.env.docker docker/.env.docker.local
+```
+
+Set (at minimum):
+
+- `APP_KEY` — generate with `docker compose run --rm app php artisan key:generate --show`.
+- Database credentials (`DB_*`). The provided MySQL service defaults to `fotospiel/secret`.
+- `STORAGE_ALERT_EMAIL` — recipient for upload failure alerts (optional).
+
+Point `docker-compose.yml` to the file you created by either renaming it to `docker/.env.docker` or adjusting the `env_file` entries.
+
+## 4. Boot the stack
+
+```bash
+docker compose up -d
+```
+
+Services started:
+
+- `app`: PHP-FPM container serving the Laravel application.
+- `web`: Nginx proxy forwarding requests to PHP-FPM on port `8080` by default (`APP_HTTP_PORT`).
+- `queue` & `media-storage-worker`: queue consumers (default + media archival).
+- `scheduler`: runs `php artisan schedule:work`.
+- `horizon` (optional, disabled unless `--profile horizon` is supplied).
+- `redis` & `mysql`.
+
+### Migrations & seeds
+
+Run once after the first boot or when deploying new schema changes:
+
+```bash
+docker compose exec app php artisan migrate --force
+docker compose exec app php artisan db:seed --class=MediaStorageTargetSeeder --force
+```
+
+If you already have data, skip the seeder or seed only new records.
+
+## 5. Queue & Horizon management
+
+Worker entrypoints live in `docs/queue-supervisor/`. The Compose services mount the same application volume so code stays in sync. Adjust concurrency by scaling services:
+
+```bash
+docker compose up -d --scale queue=2 --scale media-storage-worker=2
+```
+
+To enable Horizon (dashboard, smart balancing):
+
+```bash
+docker compose --profile horizon up -d horizon
+```
+
+## 6. Scheduler & cron jobs
+
+The compose stack ships a `scheduler` service that runs `php artisan schedule:work`, so all scheduled commands defined in `App\Console\Kernel` stay active. For upload health monitoring, keep the helper script from `cron/upload_queue_health.sh` on the host (or inside a management container) and add a cron entry:
+
+```
+*/5 * * * * /var/www/html/cron/upload_queue_health.sh
+```
+
+This wrapper logs to `storage/logs/cron-upload-queue-health.log` and executes `php artisan storage:check-upload-queues`, which in turn issues guest-facing upload alerts when queues stall or fail repeatedly. In containerised environments mount the repository so the script can reuse the same PHP binary as the app, or call the artisan command directly via `docker compose exec app php artisan storage:check-upload-queues`.
+
+The dashboard becomes available at `/horizon` and is protected by the Filament super-admin auth guard.
+
+## 6. Persistent data & volumes
+
+- `app-code` — contains the synced application, including the `storage` directory and generated assets.
+- `mysql-data` — MySQL data files.
+- `redis-data` — Redis persistence (disabled by default; change the Redis command if you want AOF snapshots).
+
+Back up the volumes before upgrades to maintain tenant media and database state.
+
+## 7. Updating the stack
+
+1. `git pull` the repository (or deploy your release branch).
+2. `docker compose build app`.
+3. `docker compose up -d`.
+4. Run migrations + seeders if required.
+5. Check logs: `docker compose logs -f app queue media-storage-worker`.
+
+Because the app image keeps the authoritative copy of the code, each container restart rsyncs fresh sources into the shared volume ensuring reliable updates without lingering artefacts.
+
+## 8. Production hardening
+
+- Terminate TLS with a dedicated reverse proxy (Traefik, Caddy, AWS ALB, etc.) in front of the `web` container.
+- Point `APP_URL` to your public domain and enable trusted proxies.
+- Externalize MySQL/Redis to managed services for better resilience.
+- Configure backups for the `storage` directories and database dumps.
+- Hook into your observability stack (e.g., ship container logs to Loki or ELK).
+
+## 9. Internal docs publishing
+
+- Build the static docs site locally or in CI via `./scripts/build-docs-site.sh` (runs `npm ci && npm run build` in `docs/site/` and outputs to `docs/site/build`).
+- The Nginx container mounts that build directory and serves it at `/internal-docs/`. History fallback is already configured in `docker/nginx/default.conf`.
+- Access is protected with HTTP Basic Auth. Update `docker/nginx/.htpasswd-docs` with real credentials (`htpasswd -c docker/nginx/.htpasswd-docs <user>`). The repo ships with a weak default for local use—never reuse it in production.
+- Optionally run the build script inside your CI pipeline before deploying so the static output is always up to date. A typical GitHub Actions job would simply call the script after checkout and upload `docs/site/build` as an artifact or rsync it to the server.
+
+With the provided configuration you can bootstrap a consistent Docker-based deployment across environments while keeping queue workers, migrations, and asset builds manageable. Adjust service definitions as needed for staging vs. production.
--- a/docs/ops/deployment/dokploy.md
+++ b/docs/ops/deployment/dokploy.md
@@ -0,0 +1,129 @@
+# Dokploy Deployment Guide
+
+Dokploy is our self-hosted PaaS for orchestrating the Fotospiel stack (Laravel app, scheduler, queue workers, Horizon, and the Photobooth FTP pipeline). This guide explains how to provision the services in Dokploy and how to wire the SuperAdmin observability widgets that now talk to the Dokploy API.
+
+## 1. Services to provision
+
+| Service | Notes |
+|---------|-------|
+| **Laravel App** | Build from this repository. Expose port 8080 (or Dokploy HTTP service). Attach the production `.env`. Health check `/up`. |
+| **Scheduler** | Clone the app container; command `php artisan schedule:work`. |
+| **Queue workers** | Use `docs/queue-supervisor/queue-worker.sh` scripts (default, media-storage, media-security). Deploy each as a dedicated Dokploy application or Docker service. |
+| **Horizon (optional)** | Run `docs/queue-supervisor/horizon.sh` for dashboard + metrics. |
+| **Redis / Database** | Use managed offerings or self-host in Dokploy. Configure network access for the app + workers. |
+| **vsftpd container** | Expose port 2121 and mount the shared Photobooth volume. |
+| **Photobooth Control Service** | Lightweight API (Go/Node/Laravel Octane) that can be redeployed together with vsftpd for ingest controls. |
+
+### Volumes
+
+Create persistent volumes inside Dokploy and mount them across the services:
+
+- `storage-app` – Laravel `storage`, uploads, compiled views.
+- `photobooth` – shared by vsftpd, the control service, and Laravel for ingest.
+- Database / Redis volumes if you self-manage those containers.
+
+## 2. Environment & secrets
+
+Every Dokploy application should include the regular Laravel secrets (see `.env.example`). Important blocks:
+
+- `APP_KEY`, `APP_URL`, `DB_*`, `CACHE_DRIVER`, `QUEUE_CONNECTION`, `MAIL_*`.
+- Photobooth integration (`PHOTOBOOTH_CONTROL_*`, `PHOTOBOOTH_FTP_*`, `PHOTOBOOTH_IMPORT_*`).
+- AWS / S3 credentials if the tenant media is stored remotely.
+
+### Dokploy integration variables
+
+Add the infrastructure observability variables to the Laravel app environment:
+
+```
+DOKPLOY_API_BASE_URL=https://dokploy.example.com/api
+DOKPLOY_API_KEY=pat_xxxxxxxxxxxxxxxxx
+DOKPLOY_WEB_URL=https://dokploy.example.com
+DOKPLOY_COMPOSE_IDS={"stack":"cmp_main","ftp":"cmp_ftp"}
+DOKPLOY_API_TIMEOUT=10
+```
+
+- `DOKPLOY_COMPOSE_IDS` ist eine JSON-Map Label → `composeId` (siehe Compose-Detailseite in Dokploy). Diese IDs steuern Widget & Buttons.
+- Optional kannst du weiterhin `DOKPLOY_APPLICATION_IDS` pflegen, falls du später einzelne Apps statt Compose-Stacks integrieren möchtest.
+- Die API benötigt Rechte für `compose.one`, `compose.loadServices`, `compose.redeploy`, `compose.stop` etc.
+
+## 3. Project & server setup
+
+1. **Register the Docker host** in Dokploy (`Servers → Add Server`). Install the Dokploy agent on the target VM.
+2. **Create a Project** (e.g., `fotospiel-prod`) to group all services.
+3. **Attach repositories** using Dokploy Git providers (GitHub / Gitea / GitLab / Bitbucket) or Docker images. Fotospiel uses the source build (Dockerfile at repo root).
+4. **Networking** – keep all services on the same internal network so they can talk to Redis/DB. Expose the public HTTP service only for the Laravel app (behind Traefik/Let’s Encrypt).
+
+## 4. Deploy applications
+
+Follow these steps for each component:
+
+1. **Laravel HTTP app**
+   - Build from the repo.
+   - `Dockerfile` already exposes port `8080`.
+   - Set branch (e.g. `main`) for automatic deployments.
+   - Add health check `/up`.
+   - Mount `storage-app` and `photobooth` volumes.
+
+2. **Scheduler**
+   - Duplicate the image.
+   - Override command: `php artisan schedule:work`.
+   - Disable HTTP exposure.
+
+3. **Queue workers**
+   - Duplicate the image.
+   - Commands:
+     - `docs/queue-supervisor/queue-worker.sh default`
+     - `docs/queue-supervisor/queue-worker.sh media-storage`
+     - `docs/queue-supervisor/queue-worker.sh media-security`
+   - Optionally create a dedicated container for Horizon using `docs/queue-supervisor/horizon.sh`.
+
+4. **vsftpd + Photobooth control**
+   - Nutze deinen bestehenden Docker-Compose-Stack (z. B. `docker-compose.dokploy.yml`) oder dedizierte Compose-Applikationen.
+   - Mount `photobooth` volume read-write.
+
+5. **Database/Redis**
+   - Dokploy can provision standard MySQL/Postgres/Redis apps. Configure credentials to match `.env`.
+
+6. **Apply migrations**
+   - Use Dokploy one-off command to run `php artisan migrate --force` on first deploy.
+   - Seed storage targets if required: `php artisan db:seed --class=MediaStorageTargetSeeder --force`.
+
+## 5. SuperAdmin observability (Dokploy API)
+
+Das SuperAdmin-Dashboard nutzt jetzt ausschließlich Compose-Endpunkte:
+
+1. **Config file** – `config/dokploy.php` liest `DOKPLOY_COMPOSE_IDS`.
+2. **Client** – `App\Services\Dokploy\DokployClient` kapselt:
+   - `GET /compose.one?composeId=...` für Meta- und Statusinfos (deploying/error/done).
+   - `GET /compose.loadServices?composeId=...` für die einzelnen Services innerhalb des Stacks.
+   - `GET /deployment.allByCompose?composeId=...` für die Deploy-Historie.
+   - `POST /compose.redeploy`, `POST /compose.deploy`, `POST /compose.stop` (Buttons im UI).
+3. **Widgets / Pages** – `DokployPlatformHealth` zeigt jeden Compose-Stack inkl. Services; die `DokployDeployments`-Seite bietet Redeploy/Stop + Audit-Log (`InfrastructureActionLog`).
+4. **Auditing** – jede Aktion wird mit User, Payload, Response & HTTP-Code in `infrastructure_action_logs` festgehalten.
+
+Only SuperAdmins should have access to these widgets. If you rotate the API key, update the `.env` and deploy the app to refresh the cache.
+
+## 6. Monitoring & alerts
+
+- Dokploy already produces container metrics and deployment logs. Surface the most important ones (CPU, memory, last deployment) through the widget using the monitoring endpoint.
+- Configure Dokploy webhooks (Deploy succeeded/failed, health alerts) to call a Laravel route that records incidents in `photobooth_metadata` or sends notifications.
+- Use Dokploy’s Slack/email integrations for infrastructure-level alerts. Application-specific alerts (e.g., ingest failures) still live inside Laravel notifications.
+
+## 7. Production readiness checklist
+
+1. Alle Compose-Stacks in Dokploy laufen mit Health Checks & Volumes.
+2. `photobooth` volume mounted for Laravel + vsftpd + control service.
+3. Database/Redis backups scheduled (Dokploy snapshot or external tooling).
+4. `.env` enthält die Dokploy-API-Credentials und `DOKPLOY_COMPOSE_IDS`.
+5. Scheduler, Worker, Horizon werden im Compose-Stack überwacht.
+6. SuperAdmin-Widget zeigt die Compose-Stacks und erlaubt Redeploy/Stop.
+7. Webhooks/alerts configured for failed deployments or unhealthy containers.
+
+With this setup the Fotospiel team can manage deployments, restarts, and metrics centrally through Dokploy while Laravel’s scheduler and workers continue to run within the same infrastructure.
+
+## 8. Internal docs publishing in Dokploy
+
+- Build the static docs site during CI/CD by running `./scripts/build-docs-site.sh`. Upload the resulting `docs/site/build` directory as part of the deployment artifact or copy it into the Dokploy server before redeploying the stack.
+- `docker-compose.dokploy.yml` mounts that directory into the Nginx container and exposes it at `/internal-docs/`, protected by HTTP Basic Auth.
+- Update `docker/nginx/.htpasswd-docs` with production credentials (use `htpasswd -c docker/nginx/.htpasswd-docs <user>` locally, then redeploy). The repository ships with a weak default only for development—always override it in Dokploy.
+- If Dokploy builds the image itself, add a post-build hook or automation step that runs the docs build and copies it into the shared storage volume before restarting the `web` service.
--- a/docs/ops/deployment/join-token-analytics.md
+++ b/docs/ops/deployment/join-token-analytics.md
@@ -0,0 +1,28 @@
+# Join Token Analytics & Alerting (SEC-GT-02)
+
+## Data Sources
+- Table `event_join_token_events` captures successes, failures, rate-limit hits, and uploads per join token.
+- Each row records route, device id, IP, HTTP status, and context for post-incident drill downs.
+- Logged automatically from `EventPublicController` for `/api/v1/events/*` and `/api/v1/gallery/*`.
+
+- Super Admin: Event resource → “Join Link / QR” modal now summarises total successes/failures, rate-limit hits, 24h volume, and last activity timestamp per token.
+- Tenant Admin: identical modal surface so operators can monitor invite health.
+
+## Alert Thresholds (initial)
+- **Rate limit spike**: >25 `token_rate_limited` entries for a token within 10 minutes → flag in monitoring (Grafana/Prometheus TODO).
+- **Failure ratio**: failure_count / success_count > 0.5 over rolling hour triggers warning for support follow-up.
+- **Inactivity**: tokens without access for >30 days should be reviewed; scheduled report TBD.
+
+Rate-limiter knobs (see `.env.example`):
+- `JOIN_TOKEN_FAILURE_LIMIT` / `JOIN_TOKEN_FAILURE_DECAY` — repeated invalid attempts before temporary block (default 10 tries per 5 min).
+- `JOIN_TOKEN_ACCESS_LIMIT` / `JOIN_TOKEN_ACCESS_DECAY` — successful request ceiling per token/IP (default 120 req per minute).
+- `JOIN_TOKEN_DOWNLOAD_LIMIT` / `JOIN_TOKEN_DOWNLOAD_DECAY` — download ceiling per token/IP (default 60 downloads per minute).
+
+## Follow-up Tasks
+1. Wire aggregated metrics into Grafana once metrics pipeline is ready (synthetic monitors pending SEC-GT-03).
+2. Implement scheduled command to email tenants a weekly digest of token activity and stale tokens.
+3. Consider anonymising device identifiers before long-term retention (privacy review).
+
+## Runbook Notes
+- Analytics table may grow quickly for high-traffic events; plan nightly prune job (keep 90 days).
+- Use `php artisan tinker` to inspect token activity: `EventJoinTokenEvent::where('event_join_token_id', $id)->latest()->limit(20)->get()`.
--- a/docs/ops/deployment/lokale-podman-adressen.md
+++ b/docs/ops/deployment/lokale-podman-adressen.md
@@ -0,0 +1,12 @@
+  Services & URLs
+
+  | Service        | URL / Port                     | Notes |
+  |----------------|--------------------------------|-------|
+  | Laravel app    | http://localhost:8000          | Default web UI; Horizon dashboard at /horizon if laravel/horizon is installed. |
+| Vite dev server| http://localhost:5173          | Hot module reload for the marketing/guest frontend. |
+  | Mailpit UI     | http://localhost:8025          | No auth; SMTP listening on port 1025. |
+  | Grafana        | http://localhost:3000          | Anonymous admin already enabled; dashboards will show Loki logs once you add Loki as a data source (URL http://loki:3100). |
+  | Loki API       | http://localhost:3100          | Used by Grafana/Promtail; direct browsing usually not needed. |
+| Portainer      | https://localhost:9443         | First visit prompts you to set an admin password; point it to /var/run/docker.sock (already mounted from the `PODMAN_SOCKET` path). |
+  | Redis          | Bound to localhost:6379        | Matches QUEUE_CONNECTION=redis. |
+  | Promtail       | Internal only (port 9080)      | Tails storage/logs and pushes to Loki. |
--- a/docs/ops/deployment/public-api-incident-playbook.md
+++ b/docs/ops/deployment/public-api-incident-playbook.md
@@ -0,0 +1,106 @@
+# Public API Incident Response Playbook (SEC-API-02)
+
+Scope: Guest-facing API endpoints that rely on join tokens and power the guest PWA plus the public gallery. This includes:
+
+- `/api/v1/events/{token}/*` (stats, tasks, uploads, photos)
+- `/api/v1/gallery/{token}/*`
+- Signed download/asset routes generated via `EventPublicController`
+
+The playbook focuses on abuse, availability loss, and leaked content.
+
+---
+
+## 1. Detection & Alerting
+
+| Signal | Where to Watch | Notes |
+| --- | --- | --- |
+| 4xx/5xx spikes | Application logs (`storage/logs/laravel.log`), centralized logging | Look for repeated `Join token access denied` / `token_rate_limited` or unexpected 5xx. |
+| Rate-limit triggers | Laravel log lines emitted from `EventPublicController::handleTokenFailure` | Contains IP + truncated token preview. |
+| CDN/WAF alerts | Reverse proxy (if enabled) | Ensure 429/403 anomalies are forwarded to incident channel. |
+| Synthetic monitors | Planned via `SEC-API-03` | Placeholder until monitors exist. |
+
+Manual check commands:
+
+```bash
+php artisan log:tail --lines=200 | grep "Join token"
+php artisan log:tail --lines=200 | grep "gallery"
+```
+
+## 2. Severity Classification
+
+| Level | Criteria | Examples |
+| --- | --- | --- |
+| SEV-1 | Wide outage (>50% error rate), confirmed data leak or malicious mass-download | Gallery downloads serving wrong event, join-token table compromised. |
+| SEV-2 | Localised outage (single tenant/event) or targeted brute force attempting to enumerate tokens | Single event returning 500, repeated `invalid_token` from single IP range. |
+| SEV-3 | Minor functional regression or cosmetic issue | Rate limit misconfiguration causing occasional 429 for legitimate users. |
+
+Escalate SEV-1/2 immediately to on-call via Slack `#incident-response` and open PagerDuty incident (if configured).
+
+## 3. Immediate Response Checklist
+
+1. **Confirm availability**  
+   - `curl -I https://app.test/api/v1/gallery/{known_good_token}`  
+   - Use tenant-provided test token to validate `/events/{token}` flow.
+2. **Snapshot logs**  
+   - Export last 15 minutes from log aggregator or `storage/logs`. Attach to incident ticket.
+3. **Assess scope**  
+   - Identify affected tenant/event IDs via log context.  
+   - Note IP addresses triggering rate limits.
+4. **Decide mitigation**  
+   - Brute force? → throttle/bock offending IPs.  
+   - Compromised token? → revoke token via Filament or `php artisan tenant:join-tokens:revoke {id}` (once command exists).  
+   - Endpoint regression? → begin rolling fix or feature flag toggle.
+
+## 4. Mitigation Tactics
+
+### 4.1 Abuse / Brute force
+- Increase rate-limiter strictness temporarily by editing `config/limiting.php` (if available) or applying runtime block in the load balancer.  
+- Use fail2ban/WAF rules to block offending IPs. For quick local action:
+  ```bash
+  sudo ufw deny from <ip_address>
+  ```
+- Consider temporarily disabling gallery download by setting `PUBLIC_GALLERY_ENABLED=false` (feature flag planned) and clearing cache.
+
+### 4.2 Token Compromise
+- Revoke specific token via Filament “Join Tokens” modal (Event → Join Tokens → revoke).  
+- Notify tenant with replacement token instructions.  
+- Audit join-token logs for additional suspicious use and consider rotating all tokens for the event.
+
+### 4.3 Internal Failure (500s)
+- Tail logs for stack traces.  
+- If due to downstream storage, fail closed: return 503 with maintenance banner while running `php artisan storage:diagnostics`.  
+- Roll back recent deployment or disable new feature flag if traced to release.
+
+## 5. Communication
+
+| Audience | Channel | Cadence |
+| --- | --- | --- |
+| Internal on-call | Slack `#incident-response`, PagerDuty | Initial alert, hourly updates. |
+| Customer Support | Slack `#support` with summary | Once per significant change (mitigation applied, issue resolved). |
+| Tenants | Email template “Public gallery disruption” (see `resources/lang/*/emails.php`) | Only for SEV-1 or impactful SEV-2 after mitigation. |
+
+Document timeline, impact, and mitigation in the incident ticket.
+
+## 6. Verification & Recovery
+
+After applying mitigation:
+
+1. Re-run test requests for affected endpoints.  
+2. Validate join-token creation/revocation via Filament.  
+3. Confirm error rates return to baseline in monitoring/dashboard.  
+4. Remove temporary firewall blocks once threat subsides.
+
+## 7. Post-Incident Actions
+
+- File RCA within 48 hours including: root cause, detection gaps, follow-up tasks (e.g., enabling synthetic monitors, adding audit fields).  
+- Update documentation if new procedures are required (`docs/prp/11-public-gallery.md`, `docs/prp/03-api.md`).  
+- Schedule backlog items for long-term fixes (e.g., better anomaly alerting, token analytics dashboards).
+
+## 8. References & Tools
+
+- Log aggregation: `storage/logs/laravel.log` (local), Stackdriver/Splunk (staging/prod).  
+- Rate limit config: `App\Providers\AppServiceProvider` → `RateLimiter::for('tenant-api')` and `EventPublicController::handleTokenFailure`.  
+- Token management UI: Filament → Events → Join Tokens.
+- Signed URL generation: `app/Http/Controllers/Api/EventPublicController` (for tracing download issues).
+
+Keep this document alongside the other deployment runbooks and review quarterly.
--- a/docs/ops/guest-notification-ops.md
+++ b/docs/ops/guest-notification-ops.md
@@ -64,5 +64,4 @@ After enabling push:
 2. Trigger a host broadcast or upload-queue alert; confirm the browser shows a native notification and that the notification drawer refreshes without polling.
 3. Temporarily stop the upload workers to create ≥5 pending assets; re-run `storage:check-upload-queues` and verify guests receive the “Uploads werden noch verarbeitet …” message.

-Document any deviations in `docs/changes/` for future regressions.
-
+Document any deviations in `docs/process/changes/` for future regressions.
--- a/docs/ops/media-storage-spec.md
+++ b/docs/ops/media-storage-spec.md
@@ -0,0 +1,99 @@
+# Media Storage Ops Specification
+
+## Purpose
+
+This document explains how customer photo uploads move through the Fotospiel platform, which services handle each stage, and what operators must monitor. It complements the PRP storage chapters by focusing on day-to-day operational expectations.
+
+## High-Level Flow
+
+1. **Upload (web / API containers)**
+   - Guest/Tenant requests hit the Laravel app container.
+   - `EventStorageManager::getHotDiskForEvent()` resolves the current *hot* storage target for the event (usually the local disk mounted at `/var/www/html/storage/app/private`).
+   - Controllers call `Storage::disk($hotDisk)->putFile("events/{event}/photos", $file)` and immediately write the original file plus a generated thumbnail to the hot disk.
+   - `EventMediaAsset` rows are created with `status = hot`, capturing disk key, relative path, checksum, size, and variant metadata.
+   - Every upload also dispatches `ProcessPhotoSecurityScan` to the `media-security` queue for antivirus and EXIF stripping.
+
+2. **Metadata & accounting**
+   - `photos` table stores user-facing references (URLs, thumbnail URLs, guest metadata).
+   - The linked `event_media_assets` records keep canonical storage information and are used later for archival/restores.
+   - `media_storage_targets` and `event_storage_assignments` tables describe which disk key is considered “hot” vs “archive” per event.
+
+3. **Asynchronous archival**
+   - The scheduler (either `php artisan schedule:work` container or host cron) runs `storage:archive-pending`.
+   - That command finds events whose galleries expired or were manually archived and dispatches `ArchiveEventMediaAssets` jobs onto the `media-storage` queue.
+   - `ArchiveEventMediaAssets` streams each file from the current disk to the archive disk resolved by `EventStorageManager::getArchiveDiskForEvent()`, updates the asset row to `status = archived`, and optionally deletes the hot copy.
+   - Archive storage usually maps to an object store bucket using the prefix `tenants/{tenant_uuid}/events/{event_uuid}/photos/{photo_uuid}/` (see `docs/prp/10-storage-media-pipeline.md`).
+
+4. **Cold storage / restore**
+   - When an event is re-opened or media needs to be rehydrated, jobs mark affected assets `restoring` and copy them back to a hot disk.
+   - Restores reuse the same `event_media_assets` bookkeeping so URLs and permissions stay consistent.
+
+## Container & Service Responsibilities
+
+| Component | Role |
+| --- | --- |
+| `app` (Laravel FPM) | Accepts uploads, writes to the hot disk, and records metadata. |
+| `media-storage-worker` | Runs `/docs/queue-supervisor/queue-worker.sh media-storage`; consumes archival/restoration jobs and copies data between disks. Shares the same `app-code` volume so it sees `/var/www/html/storage`. |
+| `queue` workers | Default queue consumers for non-storage background jobs. |
+| `media-security-worker` | Processes `ProcessPhotoSecurityScan` jobs (antivirus + EXIF scrub). |
+| `scheduler` | Runs `php artisan schedule:work`, triggering `storage:archive-pending`, `storage:monitor`, queue health checks, etc. |
+| `horizon` | Optional dashboard / supervisor for queue throughput. |
+| Redis | Queue backend for all worker containers. |
+
+## Key Commands & Jobs
+
+| Command / Job | Description |
+| --- | --- |
+| `storage:archive-pending` | Scans for expired/archived events and dispatches `ArchiveEventMediaAssets` jobs (`media-storage` queue). |
+| `storage:monitor` | Aggregates capacity/queue stats and emails alerts when thresholds are exceeded (`config/storage-monitor.php`). |
+| `media:backfill-thumbnails` | Regenerates thumbnails for existing assets; useful before enabling archival on migrated data. |
+| `ProcessPhotoSecurityScan` | Runs antivirus/EXIF stripping for a photo; default queue is `media-security`. |
+| `ArchiveEventMediaAssets` | Copies hot assets to the archive disk, updates statuses, and deletes hot copies if configured. |
+
+## Configuration Reference
+
+| Setting | Location | Notes |
+| --- | --- | --- |
+| Default disk | `.env` `FILESYSTEM_DISK` & `config/filesystems.php` | Hot uploads default to `local` (`/var/www/html/storage/app/private`). |
+| Storage targets | `media_storage_targets` table | Each row stores `key`, `driver`, and JSON config; `EventStorageManager` registers them as runtime disks. |
+| Security queue | `.env SECURITY_SCAN_QUEUE` & `config/security.php` | Defaults to `media-security`. |
+| Archive scheduling | `config/storage-monitor.php['archive']` | Controls grace days, chunk size, locking, and dispatch caps. |
+| Queue health alerts | `config/storage-monitor.php['queue_health']` | Warning/critical thresholds for `media-storage` and `media-security` queues. |
+| Container volumes | `docker-compose.yml` | `app`, workers, and scheduler share the `app-code` volume so `/var/www/html/storage` is common. |
+
+## Operational Checklist
+
+- **Before enabling archival in a new environment**
+  - Seed storage targets: `php artisan db:seed --class=MediaStorageTargetSeeder`.
+  - Run migrations so `event_media_assets` and related tables exist.
+  - Backfill existing photos into `event_media_assets` (custom script or artisan command) so archival jobs know about historical files.
+
+- **Monitoring**
+  - Watch `storage:monitor` output (email or logs) for capacity warnings on hot disks.
+  - Use Horizon or Redis metrics to verify `media-storage` queue depth; thresholds live in `config/storage-monitor.php`.
+  - Review `/var/www/html/storage/logs/storage-jobs.log` (if configured) for archival failures.
+  - Ensure `media-security` queue stays below critical thresholds so uploads aren’t blocked awaiting security scans.
+
+- **Troubleshooting uploads**
+  - Confirm hot disk is mounted and writable (`/var/www/html/storage/app/private/events/...`).
+  - Verify `media_storage_targets` contains an active `is_hot=true` entry; `EventStorageManager` falls back to default disk if none is assigned.
+  - Check Redis queue lengths; stalled `media-security` jobs prevent photos from being marked clean.
+
+- **Troubleshooting archival**
+  - Run `php artisan storage:archive-pending --event=<ID> --force` to enqueue a specific event.
+  - Tail `docker compose logs -f media-storage-worker` for copy failures.
+  - Verify archive disk credentials (e.g., S3 keys) via `media_storage_targets.config`; missing or invalid settings surface as job failures with `status=failed`.
+  - If hot copies must remain, dispatch `ArchiveEventMediaAssets` with `$deleteSource=false` (custom job call).
+
+- **Restoring media**
+  - Assign a hot storage target if none is active (`EventStorageManager::ensureAssignment($event)`).
+  - Dispatch a restore job or manually copy assets back from the archive disk, updating `event_media_assets.status` to `hot` or `restoring`.
+
+## Related Documentation
+
+- `docs/prp/10-storage-media-pipeline.md` — canonical architecture diagram for storage tiers.
+- `docs/ops/queue-workers.md` — how to run `media-storage` and `media-security` workers (scripts in `/docs/queue-supervisor/`).
+- `docs/ops/deployment/docker.md` / `docs/ops/deployment/dokploy.md` — container topology and volumes.
+- `config/security.php`, `config/storage-monitor.php`, and `config/filesystems.php` for runtime knobs.
+
+Keep this spec updated whenever the storage pipeline, queue names, or archive policies change so ops can quickly understand the flow end-to-end.
--- a/docs/ops/photobooth/README.md
+++ b/docs/ops/photobooth/README.md
@@ -0,0 +1,102 @@
+# Photobooth FTP Ingestion
+
+This guide explains how to operate the Photobooth FTP workflow end‑to‑end: provisioning FTP users for tenants, running the ingest pipeline, and exposing photobooth photos inside the Guest PWA.
+
+## Architecture Overview
+
+1. **vsftpd container** (port `2121`) accepts uploads into a shared volume (default `/var/www/storage/app/photobooth`). Each event receives isolated credentials and a dedicated directory.
+2. **Control Service** (REST) provisions FTP accounts. Laravel calls it during enable/rotate/disable actions.
+3. **Photobooth settings** (Filament SuperAdmin) define global port, rate limit, expiry grace, and Control Service connection.
+4. **Ingest command** copies uploaded files into the event’s storage disk, generates thumbnails, records `photos.ingest_source = photobooth`, and respects package quotas.
+5. **Guest PWA filter** consumes `/api/v1/events/{token}/photos?filter=photobooth` to render the “Fotobox” tab.
+
+```
+Photobooth -> FTP (vsftpd) -> photobooth disk
+       photobooth:ingest (queue/scheduler)
+            -> Event media storage (public disk/S3)
+            -> packages_usage, thumbnails, security scan
+```
+
+## Environment Variables
+
+Add the following to `.env` (already scaffolded in `.env.example`):
+
+```env
+PHOTOBOOTH_CONTROL_BASE_URL=https://control.internal/api
+PHOTOBOOTH_CONTROL_TOKEN=your-control-token
+PHOTOBOOTH_CONTROL_TIMEOUT=5
+
+PHOTOBOOTH_FTP_HOST=ftp.internal
+PHOTOBOOTH_FTP_PORT=2121
+
+PHOTOBOOTH_USERNAME_PREFIX=pb
+PHOTOBOOTH_USERNAME_LENGTH=8
+PHOTOBOOTH_PASSWORD_LENGTH=8
+
+PHOTOBOOTH_RATE_LIMIT_PER_MINUTE=20
+PHOTOBOOTH_EXPIRY_GRACE_DAYS=1
+
+PHOTOBOOTH_IMPORT_DISK=photobooth
+PHOTOBOOTH_IMPORT_ROOT=/var/www/storage/app/photobooth
+PHOTOBOOTH_IMPORT_MAX_FILES=50
+PHOTOBOOTH_ALLOWED_EXTENSIONS=jpg,jpeg,png,webp
+```
+
+### Filesystem Disk
+
+`config/filesystems.php` registers a `photobooth` disk that must point to the shared volume where vsftpd writes files. Mount the same directory inside both the FTP container and the Laravel app container.
+
+## Control Service Contract
+
+Laravel expects the Control Service to expose:
+
+```
+POST   /users                  { username, password, path, rate_limit_per_minute, expires_at, ftp_port }
+POST   /users/{username}/rotate { password, rate_limit_per_minute, expires_at }
+DELETE /users/{username}
+POST   /config                 { ftp_port, rate_limit_per_minute, expiry_grace_days }
+```
+
+Authentication is provided via `PHOTOBOOTH_CONTROL_TOKEN` (Bearer token).
+
+## Scheduler & Commands
+
+| Command | Purpose | Default schedule |
+|---------|---------|------------------|
+| `photobooth:ingest [--event=ID] [--max-files=N]` | Pulls files from the Photobooth disk and imports them into the event storage. | every 5 minutes |
+| `photobooth:cleanup-expired` | De-provisions FTP accounts after their expiry. | hourly |
+
+You can run the ingest job manually for a specific event:
+
+```bash
+php artisan photobooth:ingest --event=123 --max-files=20
+```
+
+## Tenant Admin UX
+
+Inside the Event Admin PWA, go to **Event → Fotobox-Uploads** to:
+
+1. Enable/disable the Photobooth link.
+2. Rotate credentials (max 10-char usernames, 8-char passwords).
+3. View rate limit + expiry info and copy the ftp:// link.
+
+## Guest PWA Filter
+
+The Guest gallery now exposes a “Fotobox” tab (both preview card and full gallery). API usage:
+
+```
+GET /api/v1/events/{token}/photos?filter=photobooth
+Headers: X-Device-Id (optional)
+```
+
+Response items contain `ingest_source`, allowing the frontend to toggle photobooth-only views.
+
+## Operational Checklist
+
+1. **Set env vars** from above and restart the app.
+2. **Ensure vsftpd + Control Service** are deployed; verify port 2121 and REST endpoint connectivity.
+3. **Mount shared volume** to `/var/www/storage/app/photobooth` (or update `PHOTOBOOTH_IMPORT_ROOT` + `filesystems.disks.photobooth.root`).
+4. **Run migrations** (`php artisan migrate`) to create settings/event columns.
+5. **Seed default storage target** (e.g., `MediaStorageTarget::create([... 'key' => 'public', ...])`) in non-test environments if not present.
+6. **Verify scheduler** (Horizon or cron) is running commands `photobooth:ingest` and `photobooth:cleanup-expired`.
+7. **Test end-to-end**: enable Photobooth on a staging event, upload a file via FTP, wait for ingest, and confirm it appears under the Fotobox filter in the PWA.
--- a/docs/ops/photobooth/control_service.md
+++ b/docs/ops/photobooth/control_service.md
@@ -0,0 +1,97 @@
+# Photobooth Control Service API
+
+The control service is a lightweight sidecar responsible for provisioning vsftpd accounts. Laravel talks to it via REST whenever an Event Admin enables, rotates, or disables the Photobooth feature.
+
+## Authentication
+
+- **Scheme:** Bearer token.
+- **Header:** `Authorization: Bearer ${PHOTOBOOTH_CONTROL_TOKEN}`.
+- **Timeout:** Configurable via `PHOTOBOOTH_CONTROL_TIMEOUT` (default 5 s).
+
+## Endpoints
+
+| Method & Path | Description |
+|---------------|-------------|
+| `POST /users` | Create a new FTP account for an event. |
+| `POST /users/{username}/rotate` | Rotate credentials / extend expiry for an existing user. |
+| `DELETE /users/{username}` | Remove an FTP account (called when an event disables Photobooth or expires). |
+| `POST /config` | Optionally push global config changes (port, rate-limit, expiry grace) to the control service. |
+
+### `POST /users`
+
+```json
+{
+  "username": "pbA12345",
+  "password": "F4P9K2QX",
+  "path": "tenant-slug/123",
+  "rate_limit_per_minute": 20,
+  "expires_at": "2025-06-15T22:59:59Z",
+  "ftp_port": 2121,
+  "allowed_ip_ranges": ["1.2.3.4/32"],
+  "metadata": {
+    "event_id": 123,
+    "tenant_id": 5
+  }
+}
+```
+
+**Response:** `201 Created` with `{ "ok": true }`. On failure return 4xx/5xx JSON with `error.code` + `message`.
+
+Implementation tips:
+
+- Ensure the system user or virtual user’s home directory is set to the provided `path` (prefixed with the shared Photobooth root).
+- Apply the rate limit token-bucket before writing to disk (or integrate with HAProxy).
+- Store `expires_at` and automatically disable the account when reached (in addition to Laravel’s scheduled cleanup).
+
+### `POST /users/{username}/rotate`
+
+```json
+{
+  "password": "K9M4T6QZ",
+  "rate_limit_per_minute": 20,
+  "expires_at": "2025-06-16T22:59:59Z"
+}
+```
+
+- Rotate the password atomically and respond with `{ "ok": true }`.
+- If username does not exist return `404` with a descriptive message so Laravel can re-provision.
+
+### `DELETE /users/{username}`
+
+No request body. Delete or disable the FTP account, removing access to the assigned directory.
+
+### `POST /config`
+
+Optional hook used when SuperAdmins change defaults:
+
+```json
+{
+  "ftp_port": 2121,
+  "rate_limit_per_minute": 20,
+  "expiry_grace_days": 1
+}
+```
+
+Use this to reload vsftpd or adjust proxy rules without redeploying the control service.
+
+## Error Contract
+
+Return JSON structured as:
+
+```json
+{
+  "error": {
+    "code": "user_exists",
+    "message": "Username already provisioned",
+    "context": { "username": "pbA12345" }
+  }
+}
+```
+
+Laravel treats any non-2xx as fatal and logs the payload (sans password). Prefer descriptive `code` values: `user_exists`, `user_not_found`, `rate_limit_violation`, `invalid_payload`, etc.
+
+## Observability
+
+- Emit structured logs for every create/rotate/delete with event + tenant IDs.
+- Expose `/health` so Laravel (or uptime monitors) can verify connectivity.
+- Consider metrics (e.g., Prometheus) for active accounts, rotations, and failures.
--- a/docs/ops/photobooth/ops_playbook.md
+++ b/docs/ops/photobooth/ops_playbook.md
@@ -0,0 +1,53 @@
+# Photobooth Operations Playbook
+
+Use this checklist when bringing Photobooth FTP online for a tenant or debugging ingest issues.
+
+## 1. Provisioning Flow
+
+1. **SuperAdmin config** – set defaults in Filament → Platform Management → Photobooth Settings.
+2. **Tenant enablement** – Event Admin opens the event → Fotobox-Uploads → “Photobooth aktivieren”.
+3. Laravel generates credentials and calls the control service (`POST /users`).
+4. vsftpd accepts uploads at `ftp://username:password@HOST:PORT/`.
+5. `photobooth:ingest` copies files into the hot storage disk and applies moderation/security pipelines.
+
+## 2. Troubleshooting
+
+| Symptom | Action |
+|---------|--------|
+| Tenant’s Photobooth page shows “Deaktiviert” immediately | Check `storage/logs/laravel.log` for control-service errors; re-run `photobooth:ingest --event=ID -vv`. |
+| Files remain under `/storage/app/photobooth/<tenant>/<event>` | Ensure scheduler (Horizon/cron) runs `photobooth:ingest`; run manual command to force ingestion. |
+| Photos missing from guest “Fotobox” tab | Confirm `photos.ingest_source = photobooth` and that `/api/v1/events/{token}/photos?filter=photobooth` returns data. |
+| Rate-limit complaints | Inspect control service logs; adjust `PHOTOBOOTH_RATE_LIMIT_PER_MINUTE` and re-save settings (fires `/config`). |
+| Credentials leaked/compromised | Click “Zugang neu generieren” in Event Admin; optional `php artisan photobooth:cleanup-expired --event=ID` to force deletion before expiry. |
+
+## 3. Command Reference
+
+```bash
+# Manually ingest pending files for a single event
+php artisan photobooth:ingest --event=123 --max-files=100
+
+# Check ingest for all active events (dry run)
+php artisan photobooth:ingest --max-files=10
+
+# Remove expired accounts (safe to run ad hoc)
+php artisan photobooth:cleanup-expired
+```
+
+## 4. Pre-flight Checklist for New Deployments
+
+1. `php artisan migrate`
+2. Configure `.env` Photobooth variables.
+3. Mount shared Photobooth volume in all containers (FTP + Laravel).
+4. Verify `MediaStorageTarget` records exist (hot target pointing at the hot disk).
+5. Seed baseline emotions (Photobooth ingest assigns `emotion_id` from existing rows).
+6. Confirm scheduler runs (Horizon supervisor or system cron).
+
+## 5. Incident Response
+
+1. **Identify scope** – which events/tenants are affected? Check ingestion logs for specific usernames/path.
+2. **Quarantine** – disable the Photobooth toggle for impacted events via Admin UI.
+3. **Remediate** – fix FTP/control issues, rotate credentials, run `photobooth:ingest`.
+4. **Audit** – review `photobooth_metadata` on events and `photos.ingest_source`.
+5. **Communicate** – notify tenant admins via in-app message or email template referencing incident ID.
+
+Keep this playbook updated whenever infra/process changes. PRs to `/docs/ops/photobooth` welcome.
--- a/docs/ops/queue-workers.md
+++ b/docs/ops/queue-workers.md
@@ -0,0 +1,131 @@
+## Docker Queue & Horizon Setup
+
+This directory bundles ready-to-use entrypoint scripts and deployment notes for running Fotospiel’s queue workers inside Docker containers. The examples assume you already run the main application in Docker (e.g. via `docker-compose.yml`) and share the same application image for workers. The shell scripts referenced below remain under `/docs/queue-supervisor/` so existing Dockerfile references stay valid.
+
+### 1. Prepare the application image
+
+Make sure the worker scripts are copied into the image and marked as executable:
+
+```dockerfile
+# Dockerfile
+COPY docs/queue-supervisor /var/www/html/docs/queue-supervisor
+RUN chmod +x /var/www/html/docs/queue-supervisor/*.sh
+```
+
+If you keep the project root mounted as a volume during development the `chmod` step can be skipped because the files will inherit host permissions.
+
+### 2. Queue worker containers
+
+Add one or more worker services to `docker-compose.yml`. The production compose file in the repo already defines `queue` and `media-storage-worker` services that call these scripts; the snippet below shows the essential pattern if you need to tweak scaling.
+
+```yaml
+services:
+  queue-worker:
+    image: fotospiel-app          # reuse the main app image
+    restart: unless-stopped
+    depends_on:
+      - redis                     # or your queue backend
+    environment:
+      APP_ENV: ${APP_ENV:-production}
+      QUEUE_CONNECTION: redis
+      QUEUE_TRIES: 3              # optional overrides
+      QUEUE_SLEEP: 3
+    command: >
+      /var/www/html/docs/queue-supervisor/queue-worker.sh default
+
+  media-storage-worker:
+    image: fotospiel-app
+    restart: unless-stopped
+    depends_on:
+      - redis
+    environment:
+      APP_ENV: ${APP_ENV:-production}
+      QUEUE_CONNECTION: redis
+      QUEUE_TRIES: 5
+      QUEUE_SLEEP: 5
+    command: >
+      /var/www/html/docs/queue-supervisor/queue-worker.sh media-storage
+
+  media-security-worker:
+    image: fotospiel-app
+    restart: unless-stopped
+    depends_on:
+      - redis
+    environment:
+      APP_ENV: ${APP_ENV:-production}
+      QUEUE_CONNECTION: redis
+      QUEUE_TRIES: 3
+      QUEUE_SLEEP: 5
+    command: >
+      /var/www/html/docs/queue-supervisor/queue-worker.sh media-security
+```
+
+Scale workers by increasing `deploy.replicas` (Swarm) or adding `scale` counts (Compose v2).
+
+> **Heads-up:** Guest push notifications are dispatched on the `notifications` queue. Either add that queue to the default worker (`queue-worker.sh default,notifications`) or create a dedicated worker so push jobs are consumed even when other queues are busy.
+
+### 3. Optional: Horizon container
+
+If you prefer Horizon’s dashboard and auto-balancing, add another service:
+
+```yaml
+services:
+  horizon:
+    image: fotospiel-app
+    restart: unless-stopped
+    depends_on:
+      - redis
+    environment:
+      APP_ENV: ${APP_ENV:-production}
+      QUEUE_CONNECTION: redis
+    command: >
+      /var/www/html/docs/queue-supervisor/horizon.sh
+```
+
+Expose Horizon via your web proxy and protect it with authentication (the app already guards `/horizon` behind the super admin panel login if configured).
+
+### 4. Environment variables
+
+- `QUEUE_CONNECTION` — should match the driver configured in `.env` (`redis` recommended).
+- `QUEUE_TRIES`, `QUEUE_SLEEP`, `QUEUE_TIMEOUT`, `QUEUE_MAX_TIME` — optional tuning knobs consumed by `queue-worker.sh`.
+- `STORAGE_ALERT_EMAIL` — enables upload failure notifications introduced in the new storage pipeline.
+- `SECURITY_SCAN_QUEUE` — overrides the queue name for the photo antivirus/EXIF worker (`media-security` by default).
+- Redis / database credentials must be available in the worker containers exactly like the web container.
+
+### 5. Bootstrapping reminder
+
+Before starting workers on a new environment:
+
+```bash
+php artisan migrate
+php artisan db:seed --class=MediaStorageTargetSeeder
+```
+
+Existing assets should be backfilled into `event_media_assets` with a one-off artisan command before enabling automatic archival jobs.
+
+### 6. Monitoring & logs
+
+- Containers log to STDOUT; aggregate via `docker logs` or a centralized stack.
+- Horizon users can inspect `/horizon` for queue lengths and failed jobs.
+- With plain workers run `php artisan queue:failed` (inside the container) to inspect failures and `php artisan queue:retry all` after resolving issues.
+
+### 7. Rolling updates
+
+When deploying new code:
+
+1. Build and push updated app image.
+2. Run migrations & seeders.
+3. Recreate worker/horizon containers: `docker compose up -d --force-recreate queue-worker media-storage-worker horizon`.
+4. Tail logs to confirm workers boot cleanly and start consuming jobs.
+
+### 8. Running inside Dokploy
+
+If you host Fotospiel on Dokploy:
+
+- Create separate Dokploy applications for each worker type using the same image and command snippets above (`queue-worker.sh default`, `media-storage`, etc.).
+- Attach the same environment variables and storage volumes defined for the main app.
+- Use Dokploy’s one-off command feature to run migrations or `queue:retry`.
+- Expose the Horizon service through the Dokploy HTTP proxy (or keep it internal and access via SSH tunnel).
+- Enable health checks so Dokploy restarts workers automatically if they exit unexpectedly.
+
+These services can be observed, redeployed, or reloaded from Dokploy’s dashboard and from the SuperAdmin integration powered by the Dokploy API.