# Plugin Architecture Specification ## Overview Extensible plugin system for integrating multiple AI service providers, enabling the application to work with different AI platforms (ComfyUI, RunwareAI, etc.) through a unified interface. ## Architecture Pattern **Strategy Pattern Implementation** with plugin-based architecture allowing runtime selection of AI processing backends. ``` ┌─────────────────────────────────────────────────────────────────┐ │ ImageController │ │ • Orchestrates image processing workflow │ │ • Loads appropriate plugin based on ApiProvider │ └─────────────────────┬───────────────────────────────────────────┘ │ ┌─────────────▼─────────────────────────────────────────────┐ │ PluginLoader │ │ • Discovers and instantiates plugins │ │ • Maps plugin identifiers to implementations │ └─────────────┬───────────────────────────────────────────┘ │ ┌─────────────▼─────────────────────────────────────────────┐ │ ApiPluginInterface │ │ • Defines contract for all AI plugins │ │ • Ensures consistent behavior across providers │ └─────┬─────────────────┬──────────────────────────────────┘ │ │ ┌─────▼──────┐ ┌──────▼──────┐ │ ComfyUi │ │ RunwareAi │ │ Plugin │ │ Plugin │ └────────────┘ └─────────────┘ ``` ## Core Interfaces ### ApiPluginInterface **Purpose**: Defines the contract that all AI provider plugins must implement **Location**: `app/Api/Plugins/ApiPluginInterface.php` **Methods**: #### Core Plugin Information ```php /** * Returns unique plugin identifier */ public function getIdentifier(): string /** * Returns human-readable plugin name */ public function getName(): string /** * Checks if plugin is enabled and ready for use */ public function isEnabled(): bool /** * Enables the plugin */ public function enable(): bool /** * Disables the plugin */ public function disable(): bool ``` #### Image Processing ```php /** * Main method for processing image style changes * @param Image $image Original image to transform * @param Style $style Style to apply * @return array Processing result with tracking information */ public function processImageStyleChange(Image $image, Style $style): array ``` #### Progress & Status ```php /** * Gets current processing status for image * @param string $imageUUID Image identifier * @return array Status information */ public function getStatus(string $imageUUID): array /** * Gets processing progress for image * @param string $imageUUID Image identifier * @return array Progress information (0-100 percentage) */ public function getProgress(string $imageUUID): array ``` #### Result Retrieval ```php /** * Retrieves final styled image data * @param string $promptId Provider-specific job identifier * @return string Base64 encoded image data */ public function getStyledImage(string $promptId): string ``` #### Provider-Specific Features ```php /** * Tests connection to AI service * @param array $data Connection parameters * @return bool Connection success */ public function testConnection(array $data): bool /** * Searches for available AI models * @param string $searchTerm Search query * @return array Available models */ public function searchModels(string $searchTerm): array ``` ## Plugin Implementations ### 1. ComfyUI Plugin **Plugin Identifier**: `comfyui` **Purpose**: Integration with ComfyUI web-based AI image generation platform **Configuration Requirements**: ```php // ApiProvider configuration [ 'name' => 'ComfyUI Production', 'api_url' => 'https://your-comfyui-server.com', 'plugin' => 'comfyui', 'enabled' => true ] ``` #### Key Features **Workflow-Based Processing**: - Uses JSON workflow definitions for complex AI pipelines - Supports custom node-based processing graphs - Parameter injection via placeholder replacement **WebSocket Progress Updates**: - Real-time progress monitoring via WebSocket connection - Detailed progress information (current step, completion percentage) - Live preview of intermediate results **Placeholder System**: ```json { "prompt": "__PROMPT__", "filename": "__FILENAME__", "model_id": "__MODEL_ID__" } ``` - `__PROMPT__`: Style prompt text - `__FILENAME__`: Uploaded image filename - `__MODEL_ID__`: AI model identifier #### Processing Flow 1. **Image Upload** ```php POST {api_url}/upload/image Body: multipart/form-data with image file Response: {"name": "uploaded_filename.png"} ``` 2. **Workflow Construction** ```php // Merge model and style parameters $mergedParams = array_replace_recursive($modelParams, $styleParams) // Replace placeholders $workflow = str_replace('__PROMPT__', $prompt, $workflow) $workflow = str_replace('__FILENAME__', $filename, $workflow) $workflow = str_replace('__MODEL_ID__', $modelId, $workflow) ``` 3. **Prompt Queueing** ```php POST {api_url}/prompt Body: {"prompt": $workflow} Response: {"prompt_id": "unique-job-id"} ``` 4. **Progress Monitoring** ```php // WebSocket connection ws://{server}/ws // Progress messages { "type": "progress", "data": { "prompt_id": "job-id", "value": 50, "max": 100 } } ``` 5. **Result Retrieval** ```php GET {api_url}/history/{prompt_id} // Extract image URL $imageUrl = "/view?filename={$filename}&subfolder={$subfolder}&type=output" // Download final image GET {api_url}{$imageUrl} ``` #### Error Handling - **Connection Failures**: Retry logic with exponential backoff - **Invalid Workflows**: JSON validation before submission - **Processing Timeouts**: Configurable timeout limits - **WebSocket Disconnections**: Automatic reconnection attempts --- ### 2. RunwareAI Plugin **Plugin Identifier**: `runwareai` **Purpose**: Integration with RunwareAI REST API for image style transformation **Configuration Requirements**: ```php // ApiProvider configuration [ 'name' => 'RunwareAI Production', 'api_url' => 'https://api.runwareai.com', 'token' => 'your-api-token', 'plugin' => 'runwareai', 'enabled' => true ] ``` #### Key Features **Direct API Integration**: - RESTful API communication - Bearer token authentication - Synchronous processing with immediate results **Model Search Capability**: - Search available AI models by name/type - Filter by model categories (base, checkpoint, etc.) - Real-time model availability checking **Seed Image Processing**: - Upload original image as "seed" - Apply style transformation using seed as base - Return styled result directly #### Processing Flow 1. **Image Upload** ```php POST {api_url} Headers: Authorization: Bearer {token} Body: { "taskType": "imageUpload", "taskUUID": "unique-id", "image": "data:image/png;base64,{base64_data}" } Response: { "data": [{ "imageUUID": "uploaded-image-uuid" }] } ``` 2. **Style Transformation** ```php POST {api_url} Headers: Authorization: Bearer {token} Body: { "taskType": "imageInference", "taskUUID": "unique-id", "positivePrompt": "style prompt text", "seedImage": "uploaded-image-uuid", "model": "model-id", "outputType": "base64Data" } Response: { "data": [{ "imageBase64Data": "base64-encoded-result" }] } ``` 3. **Model Search** (Optional) ```php POST {api_url} Headers: Authorization: Bearer {token} Body: { "taskType": "modelSearch", "search": "search-term", "type": "base", "category": "checkpoint", "limit": 100 } Response: { "data": [{ "results": [{ "name": "Model Name", "air": "model-id", "type": "model-type" }] }] } ``` #### Error Handling - **Authentication Failures**: Clear error messages for token issues - **Rate Limiting**: Respect API rate limits with proper delays - **Invalid Parameters**: Validation of all request parameters - **Network Issues**: Timeout handling with retry logic ## Plugin Loading Mechanism ### PluginLoader Class **Purpose**: Discovers, loads, and manages AI provider plugins **Location**: `app/Api/Plugins/PluginLoader.php` **Key Methods**: #### `getPlugin(string $pluginIdentifier, ApiProvider $apiProvider): ApiPluginInterface` - **Purpose**: Returns instantiated plugin for given identifier - **Implementation**: Dynamic class loading based on plugin identifier - **Error Handling**: Throws exception for unknown plugins #### `getAvailablePlugins(): array` - **Purpose**: Returns list of all available plugin identifiers - **Implementation**: Scans plugin directory for implementations **Plugin Discovery**: ```php // Maps plugin identifiers to class names $pluginMap = [ 'comfyui' => ComfyUi::class, 'runwareai' => RunwareAi::class, ] // Dynamic instantiation $pluginClass = $pluginMap[$identifier] ?? null; if ($pluginClass) { return new $pluginClass($apiProvider); } ``` ## Parameter Merging Strategy ### Deep Merge Algorithm **Purpose**: Combines AI model parameters with style-specific overrides **Implementation**: ```php $modelParams = $style->aiModel->parameters ?? []; $styleParams = $style->parameters ?? []; $mergedParams = array_replace_recursive($modelParams, $styleParams); ``` **Example**: ```php // Model parameters (base configuration) $modelParams = [ "workflow" => [ "nodes" => [ "1" => ["class_type" => "KSampler"], "2" => ["class_type" => "SaveImage"] ] ] ]; // Style parameters (overrides and additions) $styleParams = [ "workflow" => [ "nodes" => [ "1" => [ "inputs" => ["seed" => 12345], "class_type" => "KSamplerAdvanced" // Override ], "3" => ["class_type" => "PreviewImage"] // Addition ] ] ]; // Result: Deep merged configuration $mergedParams = [ "workflow" => [ "nodes" => [ "1" => [ "class_type" => "KSamplerAdvanced", // Overridden "inputs" => ["seed" => 12345] // Added ], "2" => ["class_type" => "SaveImage"], // Preserved "3" => ["class_type" => "PreviewImage"] // Added ] ] ] ``` ## Logging Integration ### LoggablePlugin Trait **Purpose**: Provides consistent logging across all plugins **Location**: `app/Api/Plugins/LoggablePlugin.php` **Features**: - **Structured Logging**: Consistent log format across plugins - **Log Levels**: Debug, Info, Warning, Error - **Context Information**: Plugin name, operation details - **Performance Tracking**: Execution time monitoring **Usage**: ```php class ComfyUi implements ApiPluginInterface { use LoggablePlugin; public function processImageStyleChange(Image $image, Style $style): array { $this->logInfo('Starting ComfyUI style change process', [ 'image_id' => $image->id, 'style_id' => $style->id ]); // Processing logic... $this->logInfo('ComfyUI style change completed successfully', [ 'prompt_id' => $result['prompt_id'] ]); return $result; } } ``` ## Error Handling Strategy ### Plugin-Specific Errors **ComfyUI Errors**: - **Workflow Validation**: Invalid JSON structure in workflow - **Node Connection**: Missing or invalid node connections - **Resource Limits**: GPU memory or processing queue full - **Network Timeout**: Server unreachable or slow response **RunwareAI Errors**: - **Authentication**: Invalid or expired API token - **Rate Limiting**: Too many requests per minute/hour - **Model Availability**: Requested model not available - **Image Processing**: Invalid image format or corrupted data ### Error Recovery **Retry Logic**: ```php $maxRetries = 3; $retryDelay = 1000000; // 1 second in microseconds for ($attempt = 1; $attempt <= $maxRetries; $attempt++) { try { return $this->processImageStyleChange($image, $style); } catch (RetryableException $e) { if ($attempt === $maxRetries) { throw new MaxRetriesExceededException('Processing failed after ' . $maxRetries . ' attempts'); } usleep($retryDelay * $attempt); // Exponential backoff } } ``` **Circuit Breaker Pattern**: - Track failure rates per provider - Temporarily disable failing providers - Automatic recovery testing ## Testing Strategy ### Unit Tests - **Plugin Interface**: Verify all plugins implement required methods - **Parameter Merging**: Test deep merge algorithm with various inputs - **Error Handling**: Verify proper exception throwing and recovery ### Integration Tests - **End-to-End Processing**: Test complete workflow with mock AI services - **Provider Switching**: Verify seamless transition between providers - **Error Scenarios**: Test behavior with network failures and invalid responses ### Mock Services - **ComfyUI Mock**: Simulated WebSocket and HTTP responses - **RunwareAI Mock**: Mock API responses for all endpoints - **Network Simulation**: Latency and failure injection ## Future Plugin Development ### Adding New Providers 1. **Implement Interface**: ```php class NewProvider implements ApiPluginInterface { use LoggablePlugin; public function getIdentifier(): string { return 'newprovider'; } public function getName(): string { return 'New AI Provider'; } // ... implement all required methods } ``` 2. **Register Plugin**: ```php // In PluginLoader $pluginMap['newprovider'] = NewProvider::class; ``` 3. **Add Configuration**: - Create ApiProvider record with appropriate settings - Configure authentication and endpoint details ### Plugin Enhancement Opportunities **Batch Processing Plugin**: - Process multiple images simultaneously - Queue management and progress aggregation - Resource optimization across multiple requests **Local Processing Plugin**: - Integration with local AI models (Stable Diffusion WebUI) - Reduced latency for development/testing - Offline processing capabilities **Cloud Provider Plugins**: - **AWS Bedrock**: Amazon's AI service integration - **Google Vertex AI**: Google's ML platform - **Azure Cognitive Services**: Microsoft AI services - **Replicate**: Cloud-based model hosting **Specialized Processing**: - **Upscaling Plugin**: Dedicated image enhancement - **Style Transfer Plugin**: Specialized style application - **Inpainting Plugin**: Content-aware image editing ## Configuration Management ### Provider-Specific Settings **ComfyUI Settings**: ```php // Via ApiProvider parameters (JSON) { "timeout": 180, "retry_attempts": 3, "websocket_enabled": true, "workflow_path": "/custom/workflows" } ``` **RunwareAI Settings**: ```php { "request_timeout": 60, "rate_limit_delay": 1000, "model_cache_ttl": 3600, "supported_formats": ["png", "jpg", "webp"] } ``` ### Environment-Based Configuration **Development**: - Local ComfyUI instance - Mock API responses for testing - Verbose logging enabled **Production**: - Multiple load-balanced providers - Failover between providers - Performance monitoring and alerting ## Performance Considerations ### Resource Management - **Connection Pooling**: Reuse HTTP connections where possible - **Timeout Configuration**: Appropriate timeouts for different operations - **Memory Management**: Clean up large image data after processing ### Scalability - **Concurrent Processing**: Handle multiple simultaneous requests - **Queue Management**: Prevent resource exhaustion - **Load Balancing**: Distribute requests across multiple provider instances ### Monitoring - **Performance Metrics**: Processing time, success rates, error frequencies - **Resource Usage**: Memory, CPU, network utilization - **Business Metrics**: User satisfaction, feature adoption ## Security Considerations ### Credential Management - **Encrypted Storage**: API tokens encrypted in database - **Environment Variables**: Sensitive configuration in env files - **Access Control**: Restrict plugin configuration to admin users ### Input Validation - **Image Validation**: Verify uploaded image integrity - **Parameter Sanitization**: Clean and validate all AI parameters - **Output Verification**: Ensure generated images are safe for display ### Network Security - **HTTPS Enforcement**: All external API calls over TLS - **Certificate Validation**: Verify AI service certificates - **Request Signing**: Authenticate requests where required ## Deployment Considerations ### Plugin Distribution - **Composer Packages**: Each plugin as separate package - **Auto-Discovery**: Automatic plugin registration - **Version Management**: Plugin compatibility checking ### Environment Setup - **Development**: Easy plugin development and testing - **Staging**: Pre-production validation of new plugins - **Production**: Zero-downtime plugin updates ### Backup and Recovery - **Configuration Backup**: Plugin settings and credentials - **State Preservation**: In-progress processing state - **Failover**: Automatic switching to backup providers