architecture.md

Image Upload and Hosting Service Architecture

Architecture Diagram

System Components

1. Client Interface

• API Gateway: AWS API Gateway to handle HTTP/S requests.
• Endpoints:
  • /upload for both synchronous (single image) and asynchronous (bulk) uploads.
  • /image/{id} for retrieving hosted images.
  • /status/{jobId} for polling the status of asynchronous uploads.
  • /process for defining custom processing instructions (nice-to-have).
• Request Handling:
  • Single image uploads: Processed synchronously with immediate response (either "completed" with URL or "processing" with a temporary status).
  • Bulk uploads: Processed asynchronously with a job ID returned for status polling or notification.

2. Upload Processing

• Load Balancer: Distributes incoming requests to compute instances for synchronous processing.
• Message Queue (AWS SQS):
  • Queues bulk upload tasks for asynchronous processing.
  • Ensures decoupling and scalability for high-volume uploads.
• Compute Layer (AWS EC2 or Fargate):
  • Handles synchronous uploads (single images) with immediate processing.
  • Validates image formats (JPEG, PNG, etc.) and size limits.
  • Returns "processing" status if processing exceeds a threshold (e.g., 5 seconds).
• Worker Nodes (AWS Lambda or ECS):
  • Process queued tasks for bulk uploads.
  • Perform image validation and preprocessing.

3. Image Processing

• Image Processing Service:
  • Uses libraries like ImageMagick or Pillow for resizing, cropping, and watermarking.
  • Supports custom processing instructions via JSON payloads (e.g., {"resize": {"width": 800}, "watermark": "logo.png"}).
  • Optimizes resolution/quality using compression algorithms (e.g., JPEG optimization, WebP conversion).
• Temporary Storage (AWS EFS or S3):
  • Stores images during processing to handle large payloads.

4. Storage

• Object Storage (AWS S3):
  • Stores processed images with unique identifiers.
  • Configured with lifecycle policies to manage storage costs (e.g., move to Glacier for older images).
  • Supports >10M images/day with high durability and availability.
• Metadata Database (AWS DynamoDB):
  • Stores image metadata (ID, URL, processing details, status: "processing" or "completed").
  • Enables fast retrieval, querying, and status tracking.

5. Hosting & Delivery

• Content Delivery Network (AWS CloudFront):
  • Caches images at edge locations for low-latency delivery.
  • Handles >10M image requests/day with global scalability.
• Signed URLs: Generated for secure access to private images (optional).

6. Response Notification

• Processing State Handling:
  • Synchronous Uploads:
    • If processing completes within a threshold (e.g., 5 seconds), return HTTP 200 with { "status": "completed", "imageId": "<id>", "url": "<s3-url>" }.
    • If processing takes longer, return HTTP 202 with { "status": "processing", "imageId": "<id>", "message": "Image is being processed, check /status/{imageId}" }.
  • Asynchronous (Bulk) Uploads:
    • Immediate response: HTTP 202 with { "status": "processing", "jobId": "<jobId>", "message": "Bulk upload queued, check /status/{jobId}" }.
    • Status updates available via /status/{jobId} endpoint or notifications.
• Completion Notification:
  • SNS/WebSocket: For asynchronous uploads, notifies clients when processing is complete with { "status": "completed", "jobId": "<jobId>", "urls": ["<s3-url>", ...] }.
  • Polling: Clients can query /status/{jobId} to get { "status": "processing" | "completed", "urls": ["<s3-url>", ...] }.
• Status Storage:
  • DynamoDB stores status ("processing" or "completed") for each image/job, updated by Compute Layer or Worker Nodes.

7. Monitoring & Logging

• Monitoring (AWS CloudWatch):
  • Tracks system performance, upload success rates, processing times, and status transitions.
  • Alerts for failures or prolonged "processing" states.
• Logging: Centralized logging for debugging and auditing.

How Requirements Are Fulfilled

1. Single/Bulk Image Payloads via Single Endpoint:

   • Single Endpoint (/upload):
     • Detects request type (single vs. bulk) based on payload size or metadata.
     • Single uploads: Processed synchronously by EC2/Fargate, returning "completed" or "processing" status.
     • Bulk uploads: Queued in SQS, returning "processing" status with job ID.
   • API Gateway routes requests appropriately.

2. Sync and Async Workflows:

   • Synchronous: Single uploads return "completed" (HTTP 200) with URL or "processing" (HTTP 202) with image ID if delayed.
   • Asynchronous: Bulk uploads return "processing" (HTTP 202) with job ID; clients poll /status/{jobId} or receive SNS/WebSocket notifications.

3. High Scalability (>100k Uploads, >10M Hosting):

   • Upload Scalability: SQS and auto-scaling worker nodes (Lambda/ECS) handle >100k uploads.
   • Hosting Scalability: S3 + CloudFront supports >10M requests/day with caching.
   • Database: DynamoDB scales for metadata and status tracking.

4. Image Processing Instructions (Nice-to-Have):

   • Clients specify instructions (resize, crop, watermark) in /upload payload.
   • Worker nodes and Compute Layer apply transformations using ImageMagick/Pillow.

Updated Workflow Example

1. Upload:
   • Client sends image(s) to /upload via API Gateway.
   • Single Image:
     • Routed to EC2/Fargate.
     • If processed quickly: HTTP 200 with { "status": "completed", "imageId": "<id>", "url": "<s3-url>" }.
     • If delayed: HTTP 202 with { "status": "processing", "imageId": "<id>", "message": "Check /status/{imageId}" }.
   • Bulk Upload:
     • Queued in SQS, returns HTTP 202 with { "status": "processing", "jobId": "<jobId>", "message": "Check /status/{jobId}" }.
2. Processing:
   • Synchronous: EC2/Fargate validates and processes images (resize, optimize, watermark).
   • Asynchronous: Lambda/ECS processes queued tasks.
   • Images stored in S3; metadata (ID, URL, status) saved in DynamoDB.
3. Response:
   • Synchronous: Immediate response (completed or processing).
   • Asynchronous: SNS/WebSocket notification on completion with { "status": "completed", "jobId": "<jobId>", "urls": ["<s3-url>", ...] }; clients can poll /status/{jobId} for "processing" or "completed" status.
4. Hosting:
   • Images served via CloudFront with optional signed URLs.
5. Monitoring:
   • CloudWatch tracks metrics, including "processing" state duration; alerts for issues.

Scalability & Reliability

• Auto-Scaling: EC2/Fargate and Lambda scale based on load.
• Fault Tolerance: SQS ensures no task loss; S3 provides 99.999999999% durability.
• Cost Optimization: S3 lifecycle policies and CloudFront caching reduce costs.

Diagram Description (Updated)

• Client Layer: Connects to /upload endpoint in API Gateway.
• API Gateway: Routes to Load Balancer (sync) or SQS (async).
• Load Balancer → EC2/Fargate: Handles sync processing, updates DynamoDB with "processing" or "completed".
• SQS → Lambda/ECS: Handles async processing, updates DynamoDB.
• Temporary Storage (EFS/S3): Used during processing.
• S3: Stores processed images.
• DynamoDB: Stores metadata and status ("processing" or "completed").
• CloudFront: Serves images.
• SNS/WebSocket: Notifies async job completion; supports polling via /status/{jobId}.
• CloudWatch: Monitors all components (dotted lines).
• Removed: Authentication Service (Cognito).
• Data Flow:
  • Sync: Client → API Gateway → Load Balancer → EC2 → S3/DynamoDB ("processing"/"completed") → CloudFront → Client.
  • Async: Client → API Gateway → SQS → Lambda → S3/DynamoDB ("processing"/"completed") → SNS/WebSocket → Client.

This architecture consolidates uploads to a single endpoint, removes authentication, provides clear "processing" and "completed" status responses, and ensures scalability and flexibility.