Background Worker Concurrency: Prefetching & Resource Isolation

1. Deep Architecture

By default, Celery workers prefetch tasks, fetching messages in batches. For long-running, resource-heavy tasks (like video encoding), prefetching can starve other workers. Disabling prefetching forces workers to fetch one task at a time, keeping workloads balanced.

2. The Feynman Gatekeeper

[KNOWLEDGE CHECK] Why does prefetching tasks cause issues when tasks have variable execution times, and how do you configure fair-use task distribution?

3. The Code

# config/settings.py
CELERY_WORKER_PREFETCH_MULTIPLIER = 1
CELERY_TASK_ACKS_LATE = True
# Worker command flags to set concurrency:
# celery -A config worker --concurrency=4 -Ofair

4. The Funnel

Stat Level-Up: Concurrency Master (Lvl 1).
Sanjaya Integration: Distribute video analytics tasks evenly, keeping worker nodes responsive.