fotospiel-app/docs/todo/media-streaming-upload-refactor.md

SEC-MS-02 — Streaming Upload Refactor (Requirements Draft)

Goal
Replace the current “single POST with multipart FormData” guest upload with a streaming / chunked pipeline that:

• avoids buffering entire files in PHP memory
• supports larger assets (target 25 MB originals)
• keeps antivirus/EXIF scrubbing and storage accounting intact
• exposes clear retry semantics to the guest PWA

This document captures the scope for SEC-MS-02 and feeds into implementation tickets.

---

1. Current State (Baseline)

• Upload endpoint: POST /api/v1/events/{token}/upload handled by EventPublicController::upload.
• Laravel validation enforces image|max:6144 (≈6 MB). Entire file is received via Request::file('photo').
• Storage flow: Storage::disk($hotDisk)->putFile(...) followed by synchronous thumbnail creation and event_media_assets bookkeeping.
• Device rate limiting: simple counter (guest_name = device id) per event.
• Security: join token validation + IP rate limiting; antivirus/exif cleanup handled asynchronously by ProcessPhotoSecurityScan (queued).
• Frontend: guest PWA uses fetch + FormData; progress handled by custom XHR queue for UI feedback.

Pain points:

• Upload size ceiling due to PHP post_max_size + memory usage.
• Slow devices stall the controller request; no streaming/chunk resume.
• Throttling/locks only consider completed uploads; partial data still consumes bandwidth.

---

2. Target Architecture Overview

2.1 Session-Based Chunk Upload

1. Create session

   • POST /api/v1/events/{token}/uploads → returns upload_id, upload_key, storage target, chunk size.
   • Validate join token + device limits before accepting session. Record session in new table event_upload_sessions.

2. Upload chunks

   • PUT /api/v1/events/{token}/uploads/{upload_id}/chunk with headers: Content-Range, Content-Length, Upload-Key.
   • Chunks written to hot storage stream destination (e.g. storage/app/uploads/{upload_id}/chunk_{index}) via StreamedResponse/fopen.
   • Track received ranges in session record; enforce sequential or limited parallel chunks.

3. Complete upload

   • POST /api/v1/events/{token}/uploads/{upload_id}/complete
   • Assemble chunks → single file (use stream copy to final path), compute checksum, dispatch queue jobs (AV/EXIF, thumbnail).
   • Persist photos row + event_media_assets references (mirroring current logic).

4. Abort

   • DELETE /api/v1/events/{token}/uploads/{upload_id} to clean up partial data.

2.2 Storage Strategy

• Use EventStorageManager hot disk but with temporary “staging” directory.
• After successful assembly, move to final events/{eventId}/photos/{uuid}.ext.
• For S3 targets, evaluate direct multipart upload to S3 using pre-signed URLs:
  • Option A (short-term): stream into local disk, then background job pushes to S3.
  • Option B (stretch): delegate chunk upload directly to S3 using createMultipartUpload, storing uploadId + partETags.
• Ensure staging cleanup job removes abandoned sessions (cron every hour).

2.3 Metadata & Limits

• New table event_upload_sessions fields:
  id (uuid), event_id, join_token_id, device_id, status (pending|uploading|assembling|failed|completed), total_size, received_bytes, chunk_size, expires_at, failure_reason, timestamps.
• Device/upload limits: enforce daily cap per device via session creation; consider max concurrent sessions per device/token (default 2).
• Maximum file size: 25 MB (configurable via config/media.php). Validate at complete by comparing expected vs actual bytes.

2.4 Validation & Security

• Require Upload-Key secret per session (stored hashed) to prevent hijacking.
• Join token + device validations reused; log chunk IP + UA for anomaly detection.
• Abort sessions on repeated integrity failures or mismatched Content-Range.
• Update rate limiter to consider PUT chunk endpoints separately.

2.5 API Responses & Errors

• Provide consistent JSON:
  • 201 create: { upload_id, chunk_size, expires_at }
  • chunk success: 204
  • complete: 201 { photo_id, file_path, thumbnail_path }
  • error codes: upload_limit, chunk_out_of_order, range_mismatch, session_expired.
• Document in docs/prp/03-api.md + update guest SDK.

2.6 Backend Jobs

• Assembly job (if asynchronous) ensures chunk merge is offloaded for large files; update ProcessPhotoSecurityScan to depend on final asset record.
• Add metric counters (Prometheus/Laravel events) for chunk throughput, failed sessions, average complete time.

---

3. Frontend Changes (Guest PWA)

• Replace current FormData POST with streaming uploader:
  • Request session, slice file into chunk_size (default 1 MB) using Blob.slice, upload sequentially with retry/backoff.
  • Show granular progress (bytes uploaded / total).
  • Support resume: store upload_id & received ranges in IndexedDB; on reconnect query session status from new endpoint GET /api/v1/events/{token}/uploads/{upload_id}.
• Ensure compatibility fallback: if browser lacks required APIs (e.g. old Safari), fallback to legacy single POST (size-limited) with warning.
• Update service worker/queue to pause/resume chunk uploads when offline.

---

4. Integration & Migration Tasks

1. Schema: create event_upload_sessions table + indices; optional event_upload_chunks if tracking per-part metadata.
2. Config: new entries in config/media.php for chunk size, staging path, session TTL, max size.
3. Env: add .env knobs (e.g. MEDIA_UPLOAD_CHUNK_SIZE=1048576, MEDIA_UPLOAD_MAX_SIZE=26214400).
4. Cleanup Command: php artisan media:prune-upload-sessions to purge expired sessions & staging files. Hook into cron /cron/media-prune-sessions.sh.
5. Docs: update PRP (sections 03, 10) and guest PWA README; add troubleshooting guide for chunk upload errors.
6. Testing:
   • Unit: session creation, chunk validation, assembly with mocked storage.
   • Feature: end-to-end upload success + failure (PHPUnit).
   • Playwright: simulate chunked upload with network throttling.
   • Load: ensure concurrent uploads do not exhaust disk IO.

---

5. Open Questions

• S3 Multipart vs. Local Assembly: confirm timeline for direct-to-S3; MVP may prefer local assembly to limit complexity.
• Encryption: decide whether staging chunks require at-rest encryption (likely yes if hot disk is shared).
• Quota Enforcement: should device/event caps be session-based (limit sessions) or final photo count (existing)? Combine both?
• Backward Compatibility: decide when to retire legacy endpoint; temporarily keep /upload fallback behind feature flag.

---

6. Next Steps

• Finalise design choices (S3 vs local) with Media Services.
• Break down into implementation tasks (backend API, frontend uploader, cron cleanup, observability).
• Schedule dry run in staging with large sample files (20+ MB) and monitor memory/CPU.
• Update SEC-MS-02 ticket checklist with deliverables above.