A queue is just an ordered list of work. First in, first processed, first out. Applied to video production, it means no render gets lost, no job cuts in line (unless you want it to), and your system maintains a predictable, steady output regardless of how chaotically work arrives.
Why Queues Matter for Video
Without a queue, video production is ad hoc. You render one thing, then remember you need to render another, and somewhere in between you forget a third. With a queue, every render request is captured the moment it is created. The system processes them in order, at whatever pace the hardware allows.
Queues also decouple production from consumption. The person (or automated system) creating render requests does not need to wait for the previous render to finish. They submit their request and move on. The queue absorbs bursts of demand and smooths them into a steady processing stream.
Queue Architecture Options
File-Based Queue (Simplest)
Drop JSON files into a directory. A worker process polls the directory, picks up files in timestamp order, processes them, and moves them to a "done" folder. Zero dependencies beyond the filesystem. Good for single-machine setups producing fewer than 20 videos per day.
Redis + BullMQ (Production Grade)
BullMQ provides job persistence, retry logic, priority queues, rate limiting, and a dashboard (Bull Board). Jobs survive process restarts. Failed jobs are automatically retried with exponential backoff. This is the right choice for serious production systems.
import { Queue, Worker } from 'bullmq';
const renderQueue = new Queue('video-render', {
connection: { host: '127.0.0.1', port: 6379 }
});
// Add a job
await renderQueue.add('render', {
source: '/recordings/demo.mp4',
title: 'API Demo Walkthrough'
}, {
priority: 2,
attempts: 3,
backoff: { type: 'exponential', delay: 5000 }
});
// Process jobs
const worker = new Worker('video-render', async (job) => {
await renderVideo(job.data);
}, {
concurrency: 2,
connection: { host: '127.0.0.1', port: 6379 }
});PostgreSQL SKIP LOCKED (Database-Native)
If you already run PostgreSQL, use it as a job queue. The SKIP LOCKED clause lets multiple workers pull jobs concurrently without conflicts. No additional infrastructure needed. Graphile Worker and pg-boss are Node.js libraries that wrap this pattern.
Priority and Ordering
Not all renders are equal. Client demos go before internal content. Time-sensitive changelog videos go before evergreen tutorials. A priority system lets you express this:
- Priority 1 (Critical): Client-facing content, time-sensitive releases
- Priority 2 (Normal): Scheduled content calendar items
- Priority 3 (Low): Backlog content, repurposed material
Within each priority level, jobs process in FIFO order. Higher priority jobs always process before lower priority ones, regardless of when they were submitted.
Monitoring Your Queue
Essential metrics to track:
| Metric | What It Tells You | Action Threshold |
|---|---|---|
| Queue depth | How much work is waiting | Growing consistently = add capacity |
| Processing time | How long each render takes | Increasing = investigate bottleneck |
| Failure rate | Percentage of failed renders | Above 5% = systemic issue |
| Wait time | Time from submission to processing start | Above 1 hour = add workers |
A VidNo-based pipeline naturally fits into a queue architecture. Each stage of the pipeline -- OCR, scripting, voice, render, upload -- can be modeled as a separate queue, creating a multi-stage pipeline where work flows from one queue to the next. This fine-grained approach lets you identify exactly which stage is the bottleneck.