Background Workers: Processing Video/Audio Data Streams Locally

1. Deep Architecture

Calling system binaries like FFmpeg launches separate OS processes. To prevent hanging, stdout and stderr streams must be consumed in chunks. If the system output buffer fills up before reading, it deadlocks. We run tasks asynchronously, managing CPU cores to prevent OOM errors.

2. The Feynman Gatekeeper

[KNOWLEDGE CHECK] Why does running a binary via subprocess without reading from the pipe cause the calling process to hang when processing large files?

3. The Code

# tasks.py
import subprocess
from celery import shared_task

@shared_task
def extract_metadata(file_path):
    cmd = ["ffprobe", "-v", "quiet", "-show_format", "-show_streams", "-print_format", "json", file_path]
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = proc.communicate(timeout=30)
    return stdout.decode()

4. The Funnel

Stat Level-Up: Subprocess Marshall (Lvl 1).
Sanjaya Integration: Stream raw uploaded files into local processing workers to analyze pacing, resolution, and codecs.