Long context — when to use 1M tokens (and when not to)

Engineering 8 min read· 16 Aug 2026· By Engineering

Long context — when to use 1M tokens (and when not to)

Long context isn't a substitute for RAG. Here's the framework.

Gemini 3's 1M token window is astonishing. It is not a replacement for retrieval.

Use long context when#

The corpus is small (< 500K tokens total)
Every query touches most of the corpus
You need cross-document reasoning

Use RAG when#

The corpus grows over time
Most queries touch < 5% of the corpus
You care about per-query cost

Ready to build?

Grab a key, keep your OpenAI SDK, and pay in INR.

Reranking for better RAG — cheaper than a bigger LLM

12 tactics to cut your LLM bill in half