Files

SEB Fotografie - soeren 0c83b15636 added prp

2025-11-11 21:23:00 +01:00

18 KiB

Raw Blame History

Plugin Architecture Specification

Overview

Extensible plugin system for integrating multiple AI service providers, enabling the application to work with different AI platforms (ComfyUI, RunwareAI, etc.) through a unified interface.

Architecture Pattern

Strategy Pattern Implementation with plugin-based architecture allowing runtime selection of AI processing backends.

┌─────────────────────────────────────────────────────────────────┐
│                    ImageController                              │
│  • Orchestrates image processing workflow                       │
│  • Loads appropriate plugin based on ApiProvider               │
└─────────────────────┬───────────────────────────────────────────┘
                      │
        ┌─────────────▼─────────────────────────────────────────────┐
        │            PluginLoader                                   │
        │  • Discovers and instantiates plugins                     │
        │  • Maps plugin identifiers to implementations            │
        └─────────────┬───────────────────────────────────────────┘
                      │
        ┌─────────────▼─────────────────────────────────────────────┐
        │         ApiPluginInterface                                │
        │  • Defines contract for all AI plugins                    │
        │  • Ensures consistent behavior across providers          │
        └─────┬─────────────────┬──────────────────────────────────┘
              │                 │
        ┌─────▼──────┐   ┌──────▼──────┐
        │ ComfyUi    │   │ RunwareAi   │
        │ Plugin     │   │ Plugin      │
        └────────────┘   └─────────────┘

Core Interfaces

ApiPluginInterface

Purpose: Defines the contract that all AI provider plugins must implement

Location: app/Api/Plugins/ApiPluginInterface.php

Methods:

Core Plugin Information

/**
 * Returns unique plugin identifier
 */
public function getIdentifier(): string

/**
 * Returns human-readable plugin name
 */
public function getName(): string

/**
 * Checks if plugin is enabled and ready for use
 */
public function isEnabled(): bool

/**
 * Enables the plugin
 */
public function enable(): bool

/**
 * Disables the plugin
 */
public function disable(): bool

Image Processing

/**
 * Main method for processing image style changes
 * @param Image $image Original image to transform
 * @param Style $style Style to apply
 * @return array Processing result with tracking information
 */
public function processImageStyleChange(Image $image, Style $style): array

Progress & Status

/**
 * Gets current processing status for image
 * @param string $imageUUID Image identifier
 * @return array Status information
 */
public function getStatus(string $imageUUID): array

/**
 * Gets processing progress for image
 * @param string $imageUUID Image identifier
 * @return array Progress information (0-100 percentage)
 */
public function getProgress(string $imageUUID): array

Result Retrieval

/**
 * Retrieves final styled image data
 * @param string $promptId Provider-specific job identifier
 * @return string Base64 encoded image data
 */
public function getStyledImage(string $promptId): string

Provider-Specific Features

/**
 * Tests connection to AI service
 * @param array $data Connection parameters
 * @return bool Connection success
 */
public function testConnection(array $data): bool

/**
 * Searches for available AI models
 * @param string $searchTerm Search query
 * @return array Available models
 */
public function searchModels(string $searchTerm): array

Plugin Implementations

1. ComfyUI Plugin

Plugin Identifier: comfyui

Purpose: Integration with ComfyUI web-based AI image generation platform

Configuration Requirements:

// ApiProvider configuration
[
  'name' => 'ComfyUI Production',
  'api_url' => 'https://your-comfyui-server.com',
  'plugin' => 'comfyui',
  'enabled' => true
]

Key Features

Workflow-Based Processing:

Uses JSON workflow definitions for complex AI pipelines
Supports custom node-based processing graphs
Parameter injection via placeholder replacement

WebSocket Progress Updates:

Real-time progress monitoring via WebSocket connection
Detailed progress information (current step, completion percentage)
Live preview of intermediate results

Placeholder System:

{
  "prompt": "__PROMPT__",
  "filename": "__FILENAME__",
  "model_id": "__MODEL_ID__"
}

__PROMPT__: Style prompt text
__FILENAME__: Uploaded image filename
__MODEL_ID__: AI model identifier

Processing Flow

Image Upload

POST {api_url}/upload/image
Body: multipart/form-data with image file
Response: {"name": "uploaded_filename.png"}

Workflow Construction

// Merge model and style parameters
$mergedParams = array_replace_recursive($modelParams, $styleParams)

// Replace placeholders
$workflow = str_replace('__PROMPT__', $prompt, $workflow)
$workflow = str_replace('__FILENAME__', $filename, $workflow)
$workflow = str_replace('__MODEL_ID__', $modelId, $workflow)

Prompt Queueing

POST {api_url}/prompt
Body: {"prompt": $workflow}
Response: {"prompt_id": "unique-job-id"}

Progress Monitoring

// WebSocket connection
ws://{server}/ws

// Progress messages
{
  "type": "progress",
  "data": {
    "prompt_id": "job-id",
    "value": 50,
    "max": 100
  }
}

Result Retrieval

GET {api_url}/history/{prompt_id}

// Extract image URL
$imageUrl = "/view?filename={$filename}&subfolder={$subfolder}&type=output"

// Download final image
GET {api_url}{$imageUrl}

Error Handling

Connection Failures: Retry logic with exponential backoff
Invalid Workflows: JSON validation before submission
Processing Timeouts: Configurable timeout limits
WebSocket Disconnections: Automatic reconnection attempts

2. RunwareAI Plugin

Plugin Identifier: runwareai

Purpose: Integration with RunwareAI REST API for image style transformation

Configuration Requirements:

// ApiProvider configuration
[
  'name' => 'RunwareAI Production',
  'api_url' => 'https://api.runwareai.com',
  'token' => 'your-api-token',
  'plugin' => 'runwareai',
  'enabled' => true
]

Key Features

Direct API Integration:

RESTful API communication
Bearer token authentication
Synchronous processing with immediate results

Model Search Capability:

Search available AI models by name/type
Filter by model categories (base, checkpoint, etc.)
Real-time model availability checking

Seed Image Processing:

Upload original image as "seed"
Apply style transformation using seed as base
Return styled result directly

Processing Flow

Image Upload

POST {api_url}
Headers: Authorization: Bearer {token}
Body: {
  "taskType": "imageUpload",
  "taskUUID": "unique-id",
  "image": "data:image/png;base64,{base64_data}"
}
Response: {
  "data": [{
    "imageUUID": "uploaded-image-uuid"
  }]
}

Style Transformation

POST {api_url}
Headers: Authorization: Bearer {token}
Body: {
  "taskType": "imageInference",
  "taskUUID": "unique-id",
  "positivePrompt": "style prompt text",
  "seedImage": "uploaded-image-uuid",
  "model": "model-id",
  "outputType": "base64Data"
}
Response: {
  "data": [{
    "imageBase64Data": "base64-encoded-result"
  }]
}

Model Search (Optional)

POST {api_url}
Headers: Authorization: Bearer {token}
Body: {
  "taskType": "modelSearch",
  "search": "search-term",
  "type": "base",
  "category": "checkpoint",
  "limit": 100
}
Response: {
  "data": [{
    "results": [{
      "name": "Model Name",
      "air": "model-id",
      "type": "model-type"
    }]
  }]
}

Error Handling

Authentication Failures: Clear error messages for token issues
Rate Limiting: Respect API rate limits with proper delays
Invalid Parameters: Validation of all request parameters
Network Issues: Timeout handling with retry logic

Plugin Loading Mechanism

PluginLoader Class

Purpose: Discovers, loads, and manages AI provider plugins

Location: app/Api/Plugins/PluginLoader.php

Key Methods:

`getPlugin(string $pluginIdentifier, ApiProvider $apiProvider): ApiPluginInterface`

Purpose: Returns instantiated plugin for given identifier
Implementation: Dynamic class loading based on plugin identifier
Error Handling: Throws exception for unknown plugins

`getAvailablePlugins(): array`

Purpose: Returns list of all available plugin identifiers
Implementation: Scans plugin directory for implementations

Plugin Discovery:

// Maps plugin identifiers to class names
$pluginMap = [
  'comfyui' => ComfyUi::class,
  'runwareai' => RunwareAi::class,
]

// Dynamic instantiation
$pluginClass = $pluginMap[$identifier] ?? null;
if ($pluginClass) {
  return new $pluginClass($apiProvider);
}

Parameter Merging Strategy

Deep Merge Algorithm

Purpose: Combines AI model parameters with style-specific overrides

Implementation:

$modelParams = $style->aiModel->parameters ?? [];
$styleParams = $style->parameters ?? [];
$mergedParams = array_replace_recursive($modelParams, $styleParams);

Example:

// Model parameters (base configuration)
$modelParams = [
  "workflow" => [
    "nodes" => [
      "1" => ["class_type" => "KSampler"],
      "2" => ["class_type" => "SaveImage"]
    ]
  ]
];

// Style parameters (overrides and additions)
$styleParams = [
  "workflow" => [
    "nodes" => [
      "1" => [
        "inputs" => ["seed" => 12345],
        "class_type" => "KSamplerAdvanced"  // Override
      ],
      "3" => ["class_type" => "PreviewImage"]  // Addition
    ]
  ]
];

// Result: Deep merged configuration
$mergedParams = [
  "workflow" => [
    "nodes" => [
      "1" => [
        "class_type" => "KSamplerAdvanced",  // Overridden
        "inputs" => ["seed" => 12345]       // Added
      ],
      "2" => ["class_type" => "SaveImage"],   // Preserved
      "3" => ["class_type" => "PreviewImage"] // Added
    ]
  ]
]

Logging Integration

LoggablePlugin Trait

Purpose: Provides consistent logging across all plugins

Location: app/Api/Plugins/LoggablePlugin.php

Features:

Structured Logging: Consistent log format across plugins
Log Levels: Debug, Info, Warning, Error
Context Information: Plugin name, operation details
Performance Tracking: Execution time monitoring

Usage:

class ComfyUi implements ApiPluginInterface
{
  use LoggablePlugin;

  public function processImageStyleChange(Image $image, Style $style): array
  {
    $this->logInfo('Starting ComfyUI style change process', [
      'image_id' => $image->id,
      'style_id' => $style->id
    ]);

    // Processing logic...

    $this->logInfo('ComfyUI style change completed successfully', [
      'prompt_id' => $result['prompt_id']
    ]);

    return $result;
  }
}

Error Handling Strategy

Plugin-Specific Errors

ComfyUI Errors:

Workflow Validation: Invalid JSON structure in workflow
Node Connection: Missing or invalid node connections
Resource Limits: GPU memory or processing queue full
Network Timeout: Server unreachable or slow response

RunwareAI Errors:

Authentication: Invalid or expired API token
Rate Limiting: Too many requests per minute/hour
Model Availability: Requested model not available
Image Processing: Invalid image format or corrupted data

Error Recovery

Retry Logic:

$maxRetries = 3;
$retryDelay = 1000000; // 1 second in microseconds

for ($attempt = 1; $attempt <= $maxRetries; $attempt++) {
  try {
    return $this->processImageStyleChange($image, $style);
  } catch (RetryableException $e) {
    if ($attempt === $maxRetries) {
      throw new MaxRetriesExceededException('Processing failed after ' . $maxRetries . ' attempts');
    }
    usleep($retryDelay * $attempt); // Exponential backoff
  }
}

Circuit Breaker Pattern:

Track failure rates per provider
Temporarily disable failing providers
Automatic recovery testing

Testing Strategy

Unit Tests

Plugin Interface: Verify all plugins implement required methods
Parameter Merging: Test deep merge algorithm with various inputs
Error Handling: Verify proper exception throwing and recovery

Integration Tests

End-to-End Processing: Test complete workflow with mock AI services
Provider Switching: Verify seamless transition between providers
Error Scenarios: Test behavior with network failures and invalid responses

Mock Services

ComfyUI Mock: Simulated WebSocket and HTTP responses
RunwareAI Mock: Mock API responses for all endpoints
Network Simulation: Latency and failure injection

Future Plugin Development

Adding New Providers

Implement Interface:

class NewProvider implements ApiPluginInterface
{
  use LoggablePlugin;

  public function getIdentifier(): string { return 'newprovider'; }
  public function getName(): string { return 'New AI Provider'; }
  // ... implement all required methods
}

Register Plugin:

// In PluginLoader
$pluginMap['newprovider'] = NewProvider::class;

Add Configuration:
- Create ApiProvider record with appropriate settings
- Configure authentication and endpoint details

Plugin Enhancement Opportunities

Batch Processing Plugin:

Process multiple images simultaneously
Queue management and progress aggregation
Resource optimization across multiple requests

Local Processing Plugin:

Integration with local AI models (Stable Diffusion WebUI)
Reduced latency for development/testing
Offline processing capabilities

Cloud Provider Plugins:

AWS Bedrock: Amazon's AI service integration
Google Vertex AI: Google's ML platform
Azure Cognitive Services: Microsoft AI services
Replicate: Cloud-based model hosting

Specialized Processing:

Upscaling Plugin: Dedicated image enhancement
Style Transfer Plugin: Specialized style application
Inpainting Plugin: Content-aware image editing

Configuration Management

Provider-Specific Settings

ComfyUI Settings:

// Via ApiProvider parameters (JSON)
{
  "timeout": 180,
  "retry_attempts": 3,
  "websocket_enabled": true,
  "workflow_path": "/custom/workflows"
}

RunwareAI Settings:

{
  "request_timeout": 60,
  "rate_limit_delay": 1000,
  "model_cache_ttl": 3600,
  "supported_formats": ["png", "jpg", "webp"]
}

Environment-Based Configuration

Development:

Local ComfyUI instance
Mock API responses for testing
Verbose logging enabled

Production:

Multiple load-balanced providers
Failover between providers
Performance monitoring and alerting

Performance Considerations

Resource Management

Connection Pooling: Reuse HTTP connections where possible
Timeout Configuration: Appropriate timeouts for different operations
Memory Management: Clean up large image data after processing

Scalability

Concurrent Processing: Handle multiple simultaneous requests
Queue Management: Prevent resource exhaustion
Load Balancing: Distribute requests across multiple provider instances

Monitoring

Performance Metrics: Processing time, success rates, error frequencies
Resource Usage: Memory, CPU, network utilization
Business Metrics: User satisfaction, feature adoption

Security Considerations

Credential Management

Encrypted Storage: API tokens encrypted in database
Environment Variables: Sensitive configuration in env files
Access Control: Restrict plugin configuration to admin users

Input Validation

Image Validation: Verify uploaded image integrity
Parameter Sanitization: Clean and validate all AI parameters
Output Verification: Ensure generated images are safe for display

Network Security

HTTPS Enforcement: All external API calls over TLS
Certificate Validation: Verify AI service certificates
Request Signing: Authenticate requests where required

Deployment Considerations

Plugin Distribution

Composer Packages: Each plugin as separate package
Auto-Discovery: Automatic plugin registration
Version Management: Plugin compatibility checking

Environment Setup

Development: Easy plugin development and testing
Staging: Pre-production validation of new plugins
Production: Zero-downtime plugin updates

Backup and Recovery

Configuration Backup: Plugin settings and credentials
State Preservation: In-progress processing state
Failover: Automatic switching to backup providers

18 KiB Raw Blame History

Plugin Architecture Specification

Overview

Architecture Pattern

Core Interfaces

ApiPluginInterface

Core Plugin Information

Image Processing

Progress & Status

Result Retrieval

Provider-Specific Features

Plugin Implementations

1. ComfyUI Plugin

Key Features

Processing Flow

Error Handling

2. RunwareAI Plugin

Key Features

Processing Flow

Error Handling

Plugin Loading Mechanism

PluginLoader Class

getPlugin(string $pluginIdentifier, ApiProvider $apiProvider): ApiPluginInterface

getAvailablePlugins(): array

Parameter Merging Strategy

Deep Merge Algorithm

Logging Integration

LoggablePlugin Trait

Error Handling Strategy

Plugin-Specific Errors

Error Recovery

Testing Strategy

Unit Tests

Integration Tests

Mock Services

Future Plugin Development

Adding New Providers

Plugin Enhancement Opportunities

Configuration Management

Provider-Specific Settings

Environment-Based Configuration

Performance Considerations

Resource Management

Scalability

Monitoring

Security Considerations

Credential Management

Input Validation

Network Security

Deployment Considerations

Plugin Distribution

Environment Setup

Backup and Recovery

18 KiB

Raw Blame History

`getPlugin(string $pluginIdentifier, ApiProvider $apiProvider): ApiPluginInterface`

`getAvailablePlugins(): array`