651 lines
18 KiB
Markdown
651 lines
18 KiB
Markdown
# Plugin Architecture Specification
|
|
|
|
## Overview
|
|
|
|
Extensible plugin system for integrating multiple AI service providers, enabling the application to work with different AI platforms (ComfyUI, RunwareAI, etc.) through a unified interface.
|
|
|
|
## Architecture Pattern
|
|
|
|
**Strategy Pattern Implementation** with plugin-based architecture allowing runtime selection of AI processing backends.
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ ImageController │
|
|
│ • Orchestrates image processing workflow │
|
|
│ • Loads appropriate plugin based on ApiProvider │
|
|
└─────────────────────┬───────────────────────────────────────────┘
|
|
│
|
|
┌─────────────▼─────────────────────────────────────────────┐
|
|
│ PluginLoader │
|
|
│ • Discovers and instantiates plugins │
|
|
│ • Maps plugin identifiers to implementations │
|
|
└─────────────┬───────────────────────────────────────────┘
|
|
│
|
|
┌─────────────▼─────────────────────────────────────────────┐
|
|
│ ApiPluginInterface │
|
|
│ • Defines contract for all AI plugins │
|
|
│ • Ensures consistent behavior across providers │
|
|
└─────┬─────────────────┬──────────────────────────────────┘
|
|
│ │
|
|
┌─────▼──────┐ ┌──────▼──────┐
|
|
│ ComfyUi │ │ RunwareAi │
|
|
│ Plugin │ │ Plugin │
|
|
└────────────┘ └─────────────┘
|
|
```
|
|
|
|
## Core Interfaces
|
|
|
|
### ApiPluginInterface
|
|
|
|
**Purpose**: Defines the contract that all AI provider plugins must implement
|
|
|
|
**Location**: `app/Api/Plugins/ApiPluginInterface.php`
|
|
|
|
**Methods**:
|
|
|
|
#### Core Plugin Information
|
|
```php
|
|
/**
|
|
* Returns unique plugin identifier
|
|
*/
|
|
public function getIdentifier(): string
|
|
|
|
/**
|
|
* Returns human-readable plugin name
|
|
*/
|
|
public function getName(): string
|
|
|
|
/**
|
|
* Checks if plugin is enabled and ready for use
|
|
*/
|
|
public function isEnabled(): bool
|
|
|
|
/**
|
|
* Enables the plugin
|
|
*/
|
|
public function enable(): bool
|
|
|
|
/**
|
|
* Disables the plugin
|
|
*/
|
|
public function disable(): bool
|
|
```
|
|
|
|
#### Image Processing
|
|
```php
|
|
/**
|
|
* Main method for processing image style changes
|
|
* @param Image $image Original image to transform
|
|
* @param Style $style Style to apply
|
|
* @return array Processing result with tracking information
|
|
*/
|
|
public function processImageStyleChange(Image $image, Style $style): array
|
|
```
|
|
|
|
#### Progress & Status
|
|
```php
|
|
/**
|
|
* Gets current processing status for image
|
|
* @param string $imageUUID Image identifier
|
|
* @return array Status information
|
|
*/
|
|
public function getStatus(string $imageUUID): array
|
|
|
|
/**
|
|
* Gets processing progress for image
|
|
* @param string $imageUUID Image identifier
|
|
* @return array Progress information (0-100 percentage)
|
|
*/
|
|
public function getProgress(string $imageUUID): array
|
|
```
|
|
|
|
#### Result Retrieval
|
|
```php
|
|
/**
|
|
* Retrieves final styled image data
|
|
* @param string $promptId Provider-specific job identifier
|
|
* @return string Base64 encoded image data
|
|
*/
|
|
public function getStyledImage(string $promptId): string
|
|
```
|
|
|
|
#### Provider-Specific Features
|
|
```php
|
|
/**
|
|
* Tests connection to AI service
|
|
* @param array $data Connection parameters
|
|
* @return bool Connection success
|
|
*/
|
|
public function testConnection(array $data): bool
|
|
|
|
/**
|
|
* Searches for available AI models
|
|
* @param string $searchTerm Search query
|
|
* @return array Available models
|
|
*/
|
|
public function searchModels(string $searchTerm): array
|
|
```
|
|
|
|
## Plugin Implementations
|
|
|
|
### 1. ComfyUI Plugin
|
|
|
|
**Plugin Identifier**: `comfyui`
|
|
|
|
**Purpose**: Integration with ComfyUI web-based AI image generation platform
|
|
|
|
**Configuration Requirements**:
|
|
```php
|
|
// ApiProvider configuration
|
|
[
|
|
'name' => 'ComfyUI Production',
|
|
'api_url' => 'https://your-comfyui-server.com',
|
|
'plugin' => 'comfyui',
|
|
'enabled' => true
|
|
]
|
|
```
|
|
|
|
#### Key Features
|
|
|
|
**Workflow-Based Processing**:
|
|
- Uses JSON workflow definitions for complex AI pipelines
|
|
- Supports custom node-based processing graphs
|
|
- Parameter injection via placeholder replacement
|
|
|
|
**WebSocket Progress Updates**:
|
|
- Real-time progress monitoring via WebSocket connection
|
|
- Detailed progress information (current step, completion percentage)
|
|
- Live preview of intermediate results
|
|
|
|
**Placeholder System**:
|
|
```json
|
|
{
|
|
"prompt": "__PROMPT__",
|
|
"filename": "__FILENAME__",
|
|
"model_id": "__MODEL_ID__"
|
|
}
|
|
```
|
|
- `__PROMPT__`: Style prompt text
|
|
- `__FILENAME__`: Uploaded image filename
|
|
- `__MODEL_ID__`: AI model identifier
|
|
|
|
#### Processing Flow
|
|
|
|
1. **Image Upload**
|
|
```php
|
|
POST {api_url}/upload/image
|
|
Body: multipart/form-data with image file
|
|
Response: {"name": "uploaded_filename.png"}
|
|
```
|
|
|
|
2. **Workflow Construction**
|
|
```php
|
|
// Merge model and style parameters
|
|
$mergedParams = array_replace_recursive($modelParams, $styleParams)
|
|
|
|
// Replace placeholders
|
|
$workflow = str_replace('__PROMPT__', $prompt, $workflow)
|
|
$workflow = str_replace('__FILENAME__', $filename, $workflow)
|
|
$workflow = str_replace('__MODEL_ID__', $modelId, $workflow)
|
|
```
|
|
|
|
3. **Prompt Queueing**
|
|
```php
|
|
POST {api_url}/prompt
|
|
Body: {"prompt": $workflow}
|
|
Response: {"prompt_id": "unique-job-id"}
|
|
```
|
|
|
|
4. **Progress Monitoring**
|
|
```php
|
|
// WebSocket connection
|
|
ws://{server}/ws
|
|
|
|
// Progress messages
|
|
{
|
|
"type": "progress",
|
|
"data": {
|
|
"prompt_id": "job-id",
|
|
"value": 50,
|
|
"max": 100
|
|
}
|
|
}
|
|
```
|
|
|
|
5. **Result Retrieval**
|
|
```php
|
|
GET {api_url}/history/{prompt_id}
|
|
|
|
// Extract image URL
|
|
$imageUrl = "/view?filename={$filename}&subfolder={$subfolder}&type=output"
|
|
|
|
// Download final image
|
|
GET {api_url}{$imageUrl}
|
|
```
|
|
|
|
#### Error Handling
|
|
- **Connection Failures**: Retry logic with exponential backoff
|
|
- **Invalid Workflows**: JSON validation before submission
|
|
- **Processing Timeouts**: Configurable timeout limits
|
|
- **WebSocket Disconnections**: Automatic reconnection attempts
|
|
|
|
---
|
|
|
|
### 2. RunwareAI Plugin
|
|
|
|
**Plugin Identifier**: `runwareai`
|
|
|
|
**Purpose**: Integration with RunwareAI REST API for image style transformation
|
|
|
|
**Configuration Requirements**:
|
|
```php
|
|
// ApiProvider configuration
|
|
[
|
|
'name' => 'RunwareAI Production',
|
|
'api_url' => 'https://api.runwareai.com',
|
|
'token' => 'your-api-token',
|
|
'plugin' => 'runwareai',
|
|
'enabled' => true
|
|
]
|
|
```
|
|
|
|
#### Key Features
|
|
|
|
**Direct API Integration**:
|
|
- RESTful API communication
|
|
- Bearer token authentication
|
|
- Synchronous processing with immediate results
|
|
|
|
**Model Search Capability**:
|
|
- Search available AI models by name/type
|
|
- Filter by model categories (base, checkpoint, etc.)
|
|
- Real-time model availability checking
|
|
|
|
**Seed Image Processing**:
|
|
- Upload original image as "seed"
|
|
- Apply style transformation using seed as base
|
|
- Return styled result directly
|
|
|
|
#### Processing Flow
|
|
|
|
1. **Image Upload**
|
|
```php
|
|
POST {api_url}
|
|
Headers: Authorization: Bearer {token}
|
|
Body: {
|
|
"taskType": "imageUpload",
|
|
"taskUUID": "unique-id",
|
|
"image": "data:image/png;base64,{base64_data}"
|
|
}
|
|
Response: {
|
|
"data": [{
|
|
"imageUUID": "uploaded-image-uuid"
|
|
}]
|
|
}
|
|
```
|
|
|
|
2. **Style Transformation**
|
|
```php
|
|
POST {api_url}
|
|
Headers: Authorization: Bearer {token}
|
|
Body: {
|
|
"taskType": "imageInference",
|
|
"taskUUID": "unique-id",
|
|
"positivePrompt": "style prompt text",
|
|
"seedImage": "uploaded-image-uuid",
|
|
"model": "model-id",
|
|
"outputType": "base64Data"
|
|
}
|
|
Response: {
|
|
"data": [{
|
|
"imageBase64Data": "base64-encoded-result"
|
|
}]
|
|
}
|
|
```
|
|
|
|
3. **Model Search** (Optional)
|
|
```php
|
|
POST {api_url}
|
|
Headers: Authorization: Bearer {token}
|
|
Body: {
|
|
"taskType": "modelSearch",
|
|
"search": "search-term",
|
|
"type": "base",
|
|
"category": "checkpoint",
|
|
"limit": 100
|
|
}
|
|
Response: {
|
|
"data": [{
|
|
"results": [{
|
|
"name": "Model Name",
|
|
"air": "model-id",
|
|
"type": "model-type"
|
|
}]
|
|
}]
|
|
}
|
|
```
|
|
|
|
#### Error Handling
|
|
- **Authentication Failures**: Clear error messages for token issues
|
|
- **Rate Limiting**: Respect API rate limits with proper delays
|
|
- **Invalid Parameters**: Validation of all request parameters
|
|
- **Network Issues**: Timeout handling with retry logic
|
|
|
|
## Plugin Loading Mechanism
|
|
|
|
### PluginLoader Class
|
|
|
|
**Purpose**: Discovers, loads, and manages AI provider plugins
|
|
|
|
**Location**: `app/Api/Plugins/PluginLoader.php`
|
|
|
|
**Key Methods**:
|
|
|
|
#### `getPlugin(string $pluginIdentifier, ApiProvider $apiProvider): ApiPluginInterface`
|
|
- **Purpose**: Returns instantiated plugin for given identifier
|
|
- **Implementation**: Dynamic class loading based on plugin identifier
|
|
- **Error Handling**: Throws exception for unknown plugins
|
|
|
|
#### `getAvailablePlugins(): array`
|
|
- **Purpose**: Returns list of all available plugin identifiers
|
|
- **Implementation**: Scans plugin directory for implementations
|
|
|
|
**Plugin Discovery**:
|
|
```php
|
|
// Maps plugin identifiers to class names
|
|
$pluginMap = [
|
|
'comfyui' => ComfyUi::class,
|
|
'runwareai' => RunwareAi::class,
|
|
]
|
|
|
|
// Dynamic instantiation
|
|
$pluginClass = $pluginMap[$identifier] ?? null;
|
|
if ($pluginClass) {
|
|
return new $pluginClass($apiProvider);
|
|
}
|
|
```
|
|
|
|
## Parameter Merging Strategy
|
|
|
|
### Deep Merge Algorithm
|
|
|
|
**Purpose**: Combines AI model parameters with style-specific overrides
|
|
|
|
**Implementation**:
|
|
```php
|
|
$modelParams = $style->aiModel->parameters ?? [];
|
|
$styleParams = $style->parameters ?? [];
|
|
$mergedParams = array_replace_recursive($modelParams, $styleParams);
|
|
```
|
|
|
|
**Example**:
|
|
```php
|
|
// Model parameters (base configuration)
|
|
$modelParams = [
|
|
"workflow" => [
|
|
"nodes" => [
|
|
"1" => ["class_type" => "KSampler"],
|
|
"2" => ["class_type" => "SaveImage"]
|
|
]
|
|
]
|
|
];
|
|
|
|
// Style parameters (overrides and additions)
|
|
$styleParams = [
|
|
"workflow" => [
|
|
"nodes" => [
|
|
"1" => [
|
|
"inputs" => ["seed" => 12345],
|
|
"class_type" => "KSamplerAdvanced" // Override
|
|
],
|
|
"3" => ["class_type" => "PreviewImage"] // Addition
|
|
]
|
|
]
|
|
];
|
|
|
|
// Result: Deep merged configuration
|
|
$mergedParams = [
|
|
"workflow" => [
|
|
"nodes" => [
|
|
"1" => [
|
|
"class_type" => "KSamplerAdvanced", // Overridden
|
|
"inputs" => ["seed" => 12345] // Added
|
|
],
|
|
"2" => ["class_type" => "SaveImage"], // Preserved
|
|
"3" => ["class_type" => "PreviewImage"] // Added
|
|
]
|
|
]
|
|
]
|
|
```
|
|
|
|
## Logging Integration
|
|
|
|
### LoggablePlugin Trait
|
|
|
|
**Purpose**: Provides consistent logging across all plugins
|
|
|
|
**Location**: `app/Api/Plugins/LoggablePlugin.php`
|
|
|
|
**Features**:
|
|
- **Structured Logging**: Consistent log format across plugins
|
|
- **Log Levels**: Debug, Info, Warning, Error
|
|
- **Context Information**: Plugin name, operation details
|
|
- **Performance Tracking**: Execution time monitoring
|
|
|
|
**Usage**:
|
|
```php
|
|
class ComfyUi implements ApiPluginInterface
|
|
{
|
|
use LoggablePlugin;
|
|
|
|
public function processImageStyleChange(Image $image, Style $style): array
|
|
{
|
|
$this->logInfo('Starting ComfyUI style change process', [
|
|
'image_id' => $image->id,
|
|
'style_id' => $style->id
|
|
]);
|
|
|
|
// Processing logic...
|
|
|
|
$this->logInfo('ComfyUI style change completed successfully', [
|
|
'prompt_id' => $result['prompt_id']
|
|
]);
|
|
|
|
return $result;
|
|
}
|
|
}
|
|
```
|
|
|
|
## Error Handling Strategy
|
|
|
|
### Plugin-Specific Errors
|
|
|
|
**ComfyUI Errors**:
|
|
- **Workflow Validation**: Invalid JSON structure in workflow
|
|
- **Node Connection**: Missing or invalid node connections
|
|
- **Resource Limits**: GPU memory or processing queue full
|
|
- **Network Timeout**: Server unreachable or slow response
|
|
|
|
**RunwareAI Errors**:
|
|
- **Authentication**: Invalid or expired API token
|
|
- **Rate Limiting**: Too many requests per minute/hour
|
|
- **Model Availability**: Requested model not available
|
|
- **Image Processing**: Invalid image format or corrupted data
|
|
|
|
### Error Recovery
|
|
|
|
**Retry Logic**:
|
|
```php
|
|
$maxRetries = 3;
|
|
$retryDelay = 1000000; // 1 second in microseconds
|
|
|
|
for ($attempt = 1; $attempt <= $maxRetries; $attempt++) {
|
|
try {
|
|
return $this->processImageStyleChange($image, $style);
|
|
} catch (RetryableException $e) {
|
|
if ($attempt === $maxRetries) {
|
|
throw new MaxRetriesExceededException('Processing failed after ' . $maxRetries . ' attempts');
|
|
}
|
|
usleep($retryDelay * $attempt); // Exponential backoff
|
|
}
|
|
}
|
|
```
|
|
|
|
**Circuit Breaker Pattern**:
|
|
- Track failure rates per provider
|
|
- Temporarily disable failing providers
|
|
- Automatic recovery testing
|
|
|
|
## Testing Strategy
|
|
|
|
### Unit Tests
|
|
- **Plugin Interface**: Verify all plugins implement required methods
|
|
- **Parameter Merging**: Test deep merge algorithm with various inputs
|
|
- **Error Handling**: Verify proper exception throwing and recovery
|
|
|
|
### Integration Tests
|
|
- **End-to-End Processing**: Test complete workflow with mock AI services
|
|
- **Provider Switching**: Verify seamless transition between providers
|
|
- **Error Scenarios**: Test behavior with network failures and invalid responses
|
|
|
|
### Mock Services
|
|
- **ComfyUI Mock**: Simulated WebSocket and HTTP responses
|
|
- **RunwareAI Mock**: Mock API responses for all endpoints
|
|
- **Network Simulation**: Latency and failure injection
|
|
|
|
## Future Plugin Development
|
|
|
|
### Adding New Providers
|
|
|
|
1. **Implement Interface**:
|
|
```php
|
|
class NewProvider implements ApiPluginInterface
|
|
{
|
|
use LoggablePlugin;
|
|
|
|
public function getIdentifier(): string { return 'newprovider'; }
|
|
public function getName(): string { return 'New AI Provider'; }
|
|
// ... implement all required methods
|
|
}
|
|
```
|
|
|
|
2. **Register Plugin**:
|
|
```php
|
|
// In PluginLoader
|
|
$pluginMap['newprovider'] = NewProvider::class;
|
|
```
|
|
|
|
3. **Add Configuration**:
|
|
- Create ApiProvider record with appropriate settings
|
|
- Configure authentication and endpoint details
|
|
|
|
### Plugin Enhancement Opportunities
|
|
|
|
**Batch Processing Plugin**:
|
|
- Process multiple images simultaneously
|
|
- Queue management and progress aggregation
|
|
- Resource optimization across multiple requests
|
|
|
|
**Local Processing Plugin**:
|
|
- Integration with local AI models (Stable Diffusion WebUI)
|
|
- Reduced latency for development/testing
|
|
- Offline processing capabilities
|
|
|
|
**Cloud Provider Plugins**:
|
|
- **AWS Bedrock**: Amazon's AI service integration
|
|
- **Google Vertex AI**: Google's ML platform
|
|
- **Azure Cognitive Services**: Microsoft AI services
|
|
- **Replicate**: Cloud-based model hosting
|
|
|
|
**Specialized Processing**:
|
|
- **Upscaling Plugin**: Dedicated image enhancement
|
|
- **Style Transfer Plugin**: Specialized style application
|
|
- **Inpainting Plugin**: Content-aware image editing
|
|
|
|
## Configuration Management
|
|
|
|
### Provider-Specific Settings
|
|
|
|
**ComfyUI Settings**:
|
|
```php
|
|
// Via ApiProvider parameters (JSON)
|
|
{
|
|
"timeout": 180,
|
|
"retry_attempts": 3,
|
|
"websocket_enabled": true,
|
|
"workflow_path": "/custom/workflows"
|
|
}
|
|
```
|
|
|
|
**RunwareAI Settings**:
|
|
```php
|
|
{
|
|
"request_timeout": 60,
|
|
"rate_limit_delay": 1000,
|
|
"model_cache_ttl": 3600,
|
|
"supported_formats": ["png", "jpg", "webp"]
|
|
}
|
|
```
|
|
|
|
### Environment-Based Configuration
|
|
|
|
**Development**:
|
|
- Local ComfyUI instance
|
|
- Mock API responses for testing
|
|
- Verbose logging enabled
|
|
|
|
**Production**:
|
|
- Multiple load-balanced providers
|
|
- Failover between providers
|
|
- Performance monitoring and alerting
|
|
|
|
## Performance Considerations
|
|
|
|
### Resource Management
|
|
- **Connection Pooling**: Reuse HTTP connections where possible
|
|
- **Timeout Configuration**: Appropriate timeouts for different operations
|
|
- **Memory Management**: Clean up large image data after processing
|
|
|
|
### Scalability
|
|
- **Concurrent Processing**: Handle multiple simultaneous requests
|
|
- **Queue Management**: Prevent resource exhaustion
|
|
- **Load Balancing**: Distribute requests across multiple provider instances
|
|
|
|
### Monitoring
|
|
- **Performance Metrics**: Processing time, success rates, error frequencies
|
|
- **Resource Usage**: Memory, CPU, network utilization
|
|
- **Business Metrics**: User satisfaction, feature adoption
|
|
|
|
## Security Considerations
|
|
|
|
### Credential Management
|
|
- **Encrypted Storage**: API tokens encrypted in database
|
|
- **Environment Variables**: Sensitive configuration in env files
|
|
- **Access Control**: Restrict plugin configuration to admin users
|
|
|
|
### Input Validation
|
|
- **Image Validation**: Verify uploaded image integrity
|
|
- **Parameter Sanitization**: Clean and validate all AI parameters
|
|
- **Output Verification**: Ensure generated images are safe for display
|
|
|
|
### Network Security
|
|
- **HTTPS Enforcement**: All external API calls over TLS
|
|
- **Certificate Validation**: Verify AI service certificates
|
|
- **Request Signing**: Authenticate requests where required
|
|
|
|
## Deployment Considerations
|
|
|
|
### Plugin Distribution
|
|
- **Composer Packages**: Each plugin as separate package
|
|
- **Auto-Discovery**: Automatic plugin registration
|
|
- **Version Management**: Plugin compatibility checking
|
|
|
|
### Environment Setup
|
|
- **Development**: Easy plugin development and testing
|
|
- **Staging**: Pre-production validation of new plugins
|
|
- **Production**: Zero-downtime plugin updates
|
|
|
|
### Backup and Recovery
|
|
- **Configuration Backup**: Plugin settings and credentials
|
|
- **State Preservation**: In-progress processing state
|
|
- **Failover**: Automatic switching to backup providers |