Engineering 9 min read· 23 Aug 2026· By Engineering
Batch inference at scale
Semaphores, backpressure, and how to run 100K completions overnight.
Batch jobs are the least glamorous LLM workload and often the highest ROI — enrichment, summarization, classification, tagging.
python
sem = asyncio.Semaphore(20) # 20 concurrent
async def one(item):
async with sem:
return await client.chat.completions.create(...)Cap concurrency, don't retry blindly, log every failure with the input so you can resume.