File Uploads Guide

This guide explains how to handle file uploads in JSON REST API using the FileHandlingPlugin. The system is designed to be protocol-agnostic, storage-pluggable, and schema-driven.

Overview
Quick Start
Schema Configuration
Storage Adapters
Protocol Configuration
File Validation
Complete Examples
Troubleshooting

Overview

The file handling system consists of three main components:

FileHandlingPlugin - Orchestrates file detection and processing
Protocol Detectors - Parse files from different protocols (HTTP, Express)
Storage Adapters - Save files to different backends (local, S3, etc.)

How It Works

You define file fields in your schema with type: 'file'
Protocol plugins detect and parse multipart uploads
FileHandlingPlugin validates and processes files
Storage adapters save files and return URLs
File fields are replaced with URLs in your data

Quick Start

import { Api } from 'hooked-api';
import { RestApiPlugin, FileHandlingPlugin, ExpressPlugin } from 'json-rest-api';
import { LocalStorage } from 'json-rest-api/plugins/storage/local-storage.js';

// Create API
const api = new Api({ name: 'my-api', version: '1.0.0' });

// Create storage
const storage = new LocalStorage({
  directory: './uploads',
  baseUrl: 'http://localhost:3000/uploads'
});

// Use plugins (order matters!)
api.use(RestApiPlugin);
api.use(FileHandlingPlugin);
api.use(ExpressPlugin);

// Define schema with file field
api.addScope('images', {
  schema: {
    title: { type: 'string', required: true },
    file: { 
      type: 'file',
      storage: storage,
      accepts: ['image/*'],
      maxSize: '10mb'
    }
  }
});

Schema Configuration

File fields are defined in your scope schema with type: 'file':

api.addScope('documents', {
  schema: {
    // Regular fields
    title: { type: 'string', required: true },
    description: { type: 'string' },
    
    // File field
    attachment: {
      type: 'file',
      storage: myStorage,        // Required: storage adapter instance
      accepts: ['*'],            // Optional: accepted mime types (default: ['*'])
      maxSize: '50mb',           // Optional: max file size
      required: false            // Optional: is field required? (default: false)
    }
  }
});

File Field Options

type: Must be 'file'
storage: Storage adapter instance (required)
accepts: Array of accepted MIME types
- ['*'] - Accept any file type (default)
- ['image/*'] - Accept any image
- ['image/jpeg', 'image/png'] - Accept specific types
- ['application/pdf', 'text/*'] - Mix specific and wildcard
maxSize: Maximum file size
- '10mb', '1.5gb', '500kb' - Human readable format
- Number of bytes also supported
required: Whether the file is required

Storage Adapters

Storage adapters handle where and how files are saved. The library includes two built-in adapters.

Storage Adapter Comparison

Feature	LocalStorage	S3Storage
Production Ready	✅ Yes	⚠️ Mock only
Filename Strategies	4 (hash, timestamp, original, custom)	1 (hash only)
Path Traversal Protection	✅ Full	✅ N/A
Extension Whitelist	✅ Yes	❌ No
Duplicate Handling	✅ Yes	✅ Automatic
Custom Naming	✅ Yes	❌ No
Best For	Local file storage	Cloud storage

S3Storage

Saves files to Amazon S3 or S3-compatible storage:

import { S3Storage } from 'json-rest-api/plugins/storage/s3-storage.js';

const s3Storage = new S3Storage({
  bucket: 'my-uploads',                // S3 bucket name (required)
  region: 'us-east-1',                 // AWS region (default: 'us-east-1')
  prefix: 'uploads/',                  // Path prefix in bucket (default: '')
  acl: 'public-read',                  // Access control (default: 'public-read')
  mockMode: false                      // Use mock mode? (default: true)
});

Filename Handling:

Always generates random hash + extension (e.g., uploads/a7f8d9e2b4c6e1f3.jpg)
Original filenames are never used for security

Note: The included S3Storage is a mock implementation for demonstration. For production use, you’ll need to implement the actual AWS SDK calls.

LocalStorage

Saves files to the local filesystem with secure filename handling:

import { LocalStorage } from 'json-rest-api/plugins/storage/local-storage.js';

const localStorage = new LocalStorage({
  directory: './uploads',              // Where to save files
  baseUrl: '/uploads',                 // Public URL prefix
  nameStrategy: 'hash',                // Filename strategy (see below)
  preserveExtension: true,             // Keep file extensions? (default: true)
  allowedExtensions: ['.jpg', '.png'], // Extension whitelist (optional)
  maxFilenameLength: 255,              // Max filename length
  nameGenerator: async (file) => {...} // Custom name generator (optional)
});

Filename Strategies:

'hash' (default) - Cryptographically secure random names

nameStrategy: 'hash'
// Result: "a7f8d9e2b4c6e1f3.jpg"

'timestamp' - Timestamp with random suffix (sortable)

nameStrategy: 'timestamp'
// Result: "1672531200000_a8f9.pdf"

'original' - Sanitized original filename (user-friendly)

nameStrategy: 'original'
// "My Photo!.jpg" → "My_Photo_.jpg"
// Duplicates → "My_Photo_1.jpg", "My_Photo_2.jpg"

'custom' - Your own naming logic

nameStrategy: 'custom',
nameGenerator: async (file) => {
  const userId = file.metadata?.userId || 'anonymous';
  return `user_${userId}_${Date.now()}`;
}
// Result: "user_12345_1672531200000.jpg"

Security Features:

Path traversal protection (removes .. and /)
Control character filtering
Extension validation against whitelist
Automatic duplicate handling
MIME type to extension mapping

Custom Storage Adapters

Create your own storage adapter by implementing the required interface:

class MyCustomStorage {
  async upload(file) {
    // file object contains:
    // - filename: original filename
    // - mimetype: MIME type
    // - size: size in bytes
    // - data: Buffer with file contents
    // - filepath: temp file path (if using formidable)
    // - cleanup: async function to cleanup temp files
    
    // Save the file somewhere
    const url = await saveFileSomewhere(file);
    
    // Return the public URL
    return url;
  }
  
  async delete(url) {
    // Optional: implement file deletion
    await deleteFileSomewhere(url);
  }
}

Protocol Configuration

Different protocols have different configuration options for file parsing.

ExpressPlugin Configuration

The Express plugin supports multiple file parsers:

api.use(ExpressPlugin, {
  // Choose parser: 'busboy', 'formidable', or a function
  fileParser: 'busboy',
  
  // Parser-specific options
  fileParserOptions: {
    // For busboy
    limits: {
      fileSize: 10 * 1024 * 1024,  // 10MB max file size
      files: 5,                     // Max 5 files per request
      fields: 20,                   // Max 20 non-file fields
      parts: 25                     // Max 25 total parts
    }
  },
  
  // Or use formidable
  // fileParser: 'formidable',
  // fileParserOptions: {
  //   uploadDir: './temp',        // Temp directory
  //   keepExtensions: true,       // Keep file extensions
  //   maxFileSize: 200 * 1024 * 1024  // 200MB max
  // }
  
  // Disable file uploads entirely
  // enableFileUploads: false
});

Using Express Middleware

For advanced use cases, you can use Express middleware for file handling:

import multer from 'multer';

const upload = multer({ dest: 'uploads/' });

api.use(ExpressPlugin, {
  middleware: {
    beforeScope: {
      // Add multer to specific scope
      images: [upload.single('file')],
      
      // Multiple files for another scope
      gallery: [upload.array('photos', 10)]
    }
  },
  
  // Disable built-in file handling since we're using multer
  enableFileUploads: false
});

HttpPlugin Configuration

The HTTP plugin has similar configuration:

api.use(HttpPlugin, {
  // Choose parser: 'busboy', 'formidable', or a function
  fileParser: 'formidable',
  
  fileParserOptions: {
    uploadDir: './uploads/temp',
    keepExtensions: true,
    maxFileSize: 100 * 1024 * 1024  // 100MB
  },
  
  // Other HTTP options
  port: 3000,
  basePath: '/api'
});

Custom File Parsers

You can provide a custom file parser:

api.use(HttpPlugin, {
  fileParser: (options) => ({
    name: 'my-custom-parser',
    detect: (params) => {
      // Return true if this parser can handle the request
      const req = params._httpReq;
      return req.headers['content-type']?.includes('multipart/form-data');
    },
    parse: async (params) => {
      // Parse the request and return { fields, files }
      const req = params._httpReq;
      const { fields, files } = await myCustomParser(req);
      return { fields, files };
    }
  })
});

File Validation

The FileHandlingPlugin automatically validates files based on schema configuration.

MIME Type Validation

// Accept only images
file: {
  type: 'file',
  storage: localStorage,
  accepts: ['image/*']
}

// Accept specific types
document: {
  type: 'file',
  storage: s3Storage,
  accepts: ['application/pdf', 'application/msword', 'text/plain']
}

// Accept anything (not recommended)
attachment: {
  type: 'file',
  storage: localStorage,
  accepts: ['*']
}

Size Validation

// Human-readable format
avatar: {
  type: 'file',
  storage: localStorage,
  maxSize: '5mb'
}

// Supports: b, kb, mb, gb
largeFile: {
  type: 'file',
  storage: s3Storage,
  maxSize: '1.5gb'
}

Required Files

// This file must be provided
document: {
  type: 'file',
  storage: s3Storage,
  required: true
}

Validation Errors

When validation fails, you’ll get appropriate error responses:

{
  "errors": [{
    "status": "422",
    "title": "Validation Error",
    "detail": "Invalid file type for field 'avatar'",
    "source": {
      "pointer": "/data/attributes/avatar"
    }
  }]
}

Complete Examples

Basic Image Upload

import { Api } from 'hooked-api';
import { RestApiPlugin, FileHandlingPlugin, ExpressPlugin } from 'json-rest-api';
import { LocalStorage } from 'json-rest-api/plugins/storage/local-storage.js';
import express from 'express';

// Setup
const api = new Api({ name: 'image-api', version: '1.0.0' });
const storage = new LocalStorage({
  directory: './uploads/images',
  baseUrl: 'http://localhost:3000/uploads/images'
});

// Plugins
api.use(RestApiPlugin);
api.use(FileHandlingPlugin);
api.use(ExpressPlugin);

// Schema
api.addScope('photos', {
  schema: {
    caption: { type: 'string', required: true },
    image: {
      type: 'file',
      storage: storage,
      accepts: ['image/jpeg', 'image/png'],
      maxSize: '10mb',
      required: true
    }
  }
});

// Express app
const app = express();
app.use('/uploads', express.static('./uploads'));
api.express.mount(app);

// HTML form for testing
app.get('/', (req, res) => {
  res.send(`
    <form action="/api/photos" method="POST" enctype="multipart/form-data">
      <input name="caption" placeholder="Caption" required>
      <input name="image" type="file" accept="image/*" required>
      <button type="submit">Upload Photo</button>
    </form>
  `);
});

app.listen(3000).on('error', (err) => {
  console.error('Failed to start server:', err);
  process.exit(1)
});

Multiple Storage Backends

// Different storage for different fields
api.addScope('articles', {
  schema: {
    title: { type: 'string', required: true },
    content: { type: 'string', required: true },
    
    // Featured image goes to S3
    featuredImage: {
      type: 'file',
      storage: s3Storage,
      accepts: ['image/*'],
      maxSize: '20mb'
    },
    
    // Attachments stay local
    attachment: {
      type: 'file',
      storage: localStorage,
      accepts: ['application/pdf', 'application/zip'],
      maxSize: '100mb'
    }
  }
});

Filename Handling Examples

Different strategies for different use cases:

import { LocalStorage } from 'json-rest-api/plugins/storage/local-storage.js';

// User avatars - use hash for security and deduplication
const avatarStorage = new LocalStorage({
  directory: './uploads/avatars',
  baseUrl: '/uploads/avatars',
  nameStrategy: 'hash'
});

// Documents - use timestamp for sorting
const documentStorage = new LocalStorage({
  directory: './uploads/documents',
  baseUrl: '/uploads/documents',
  nameStrategy: 'timestamp'
});

// User downloads - preserve original names
const downloadStorage = new LocalStorage({
  directory: './uploads/downloads',
  baseUrl: '/uploads/downloads',
  nameStrategy: 'original',
  maxFilenameLength: 100
});

// High security - no extensions
const secureStorage = new LocalStorage({
  directory: './uploads/secure',
  baseUrl: '/uploads/secure',
  nameStrategy: 'hash',
  preserveExtension: false,  // All files saved as .bin
  allowedExtensions: ['.pdf', '.doc', '.docx']  // Still validates input
});

// Organized by date
const organizedStorage = new LocalStorage({
  directory: './uploads',
  baseUrl: '/uploads',
  nameStrategy: 'custom',
  nameGenerator: async (file) => {
    const date = new Date();
    const dateDir = `${date.getFullYear()}/${String(date.getMonth() + 1).padStart(2, '0')}`;
    return `${dateDir}/${crypto.randomBytes(16).toString('hex')}`;
  }
});
// Saves as: "2024/01/a7f8d9e2b4c6e1f3.jpg"

Using cURL

# Upload with cURL
curl -X POST http://localhost:3000/api/photos \
  -F "caption=Beautiful sunset" \
  -F "image=@/path/to/sunset.jpg"

# Multiple files
curl -X POST http://localhost:3000/api/articles \
  -F "title=My Article" \
  -F "content=Article content here" \
  -F "featuredImage=@/path/to/hero.jpg" \
  -F "attachment=@/path/to/document.pdf"

Programmatic Upload

// Using fetch with FormData
const formData = new FormData();
formData.append('caption', 'My photo');
formData.append('image', fileInput.files[0]);

const response = await fetch('/api/photos', {
  method: 'POST',
  body: formData
});

const result = await response.json();
console.log('Uploaded:', result.data.attributes.image); // URL of uploaded file

Troubleshooting

Common Issues

1. “Busboy not available, file uploads disabled”

Install the peer dependency:

npm install busboy

2. “No storage configured for file field”

Make sure you’ve set the storage property:

file: {
  type: 'file',
  storage: myStorage  // This is required!
}

3. Files not being detected

Check that:

The FileHandlingPlugin is loaded AFTER RestApiPlugin
Your protocol plugin has file uploads enabled
The request has proper multipart headers
You’re using the correct form encoding

<!-- HTML forms need this -->
<form enctype="multipart/form-data">

4. “File too large” errors

Check both:

Schema maxSize configuration
Parser limits in plugin options

// Both limits apply!
api.use(ExpressPlugin, {
  fileParserOptions: {
    limits: { fileSize: 10 * 1024 * 1024 }  // 10MB parser limit
  }
});

api.addScope('images', {
  schema: {
    photo: {
      type: 'file',
      maxSize: '5mb'  // 5MB schema limit (lower wins)
    }
  }
});

Debug Mode

Enable debug logging to see what’s happening:

const api = new Api({
  name: 'my-api',
  version: '1.0.0',
  logLevel: 'debug'  // or 'trace' for more detail
});

Testing File Uploads

Use the included example to test your setup:

// Run the example
node ./node_modules/json-rest-api/examples/file-upload-example.js

Then visit http://localhost:3000 to see the test forms.

Best Practices

Always validate file types - Don’t use accepts: ['*'] in production
Set reasonable size limits - Prevent abuse and server overload
Use appropriate storage - Local for small files, S3 for large/many files
Clean up temp files - Storage adapters should handle cleanup
Serve files separately - Don’t serve uploaded files through your API
Validate file contents - Consider virus scanning for user uploads
Use CDN for images - Serve uploaded images through a CDN in production

Security Considerations

Filename Security

Never trust user filenames - Always sanitize or generate new names

// BAD - Direct use of user filename
const filename = file.originalname;
   
// GOOD - Generate secure name
const filename = crypto.randomBytes(16).toString('hex') + path.extname(file.originalname);

Prevent path traversal - Remove dangerous characters

// Dangerous filenames to watch for:
// "../../../etc/passwd"
// "..\\..\\windows\\system32\\config\\sam"
// "uploads/../../../index.js"
   
// LocalStorage handles this automatically

Extension validation - Whitelist allowed extensions

// Use LocalStorage with whitelist
const storage = new LocalStorage({
  allowedExtensions: ['.jpg', '.jpeg', '.png', '.pdf']
});

Consider removing extensions entirely for sensitive files

const highSecurityStorage = new LocalStorage({
  nameStrategy: 'hash',
  preserveExtension: false  // All files become .bin
});

General File Security

Validate MIME types - But remember they can be spoofed
Check file contents - Use libraries like file-type for verification
Limit upload sizes - Prevent denial of service
Store files outside web root - Prevent direct execution
Use virus scanning - For user-uploaded content
Set proper permissions - Uploaded files shouldn’t be executable
Serve files with proper headers - Use Content-Disposition for downloads

Next Steps

Implement production S3 storage with actual AWS SDK
Add image processing (thumbnails, resizing)
Implement virus scanning for uploads
Add progress tracking for large files
Create a chunked upload system for very large files