Managing Tokens and Context Windows for Large Language Models
Why answers from a language model get cut off mid-sentence, and how to budget what you send and what you ask for so they don't.
On this page
If you've shipped anything on top of a language model, you've already been managing tokens, whether or not you called it that. Every request spends three kinds: the input you send, the output the model returns, and, on reasoning models, the hidden thinking in between. Get the budget wrong and you see answers that stop mid-sentence, latency spikes and a bill nobody can forecast. Get it right and the application is reliable, fast and predictable, which is what your users and whoever pays the bill will judge you on.
The whole essay reduces to one rule, and I'll state it now so you can hold the rest against it: treat tokens the way you already treat CPU time and memory. Budget them per request, cap them, and measure what you spend.
Here's the route. First, what a token is, and why the context window and the output cap are two separate limits, because every technique later depends on that distinction. Second, why the budget still matters when providers advertise million-token context windows. Third, the techniques, from doing nothing to computing a budget per request, with working code, the telemetry that tells you whether it's working, and what to return when the budget runs short.
What a token is
A language model doesn't read words. It reads short runs of characters, spaces and punctuation included, and each run is a token. For English text a rough rule is one token for every four characters, or about three quarters of a word. Treat that as an estimate: the real count depends on the model family and the language, and when the count matters, run the model's own tokeniser rather than guess (Hugging Face tokenizer docs).
Tokenisation, the step that turns text into tokens, also differs between providers. OpenAI's tiktoken and the SentencePiece family split the same string differently, so if you route requests across models, don't assume a string costs the same everywhere. Count it per model.
Input, output and reasoning tokens
Input tokens are what you send: the system prompt, the conversation so far, the user's message, and any documents you retrieved for it. They're called input because, from the model's side, you're handing them over.
Output tokens are what the model returns, the visible answer.
Reasoning tokens are the hidden thinking a reasoning model does before it answers. You never see that text, but it appears in the usage metadata and you pay for it. Azure and OpenAI report it as completion_tokens_details.reasoning_tokens (Microsoft Learn on reasoning models). When we compute a budget later, all three kinds sit in the same equation.
The context window and the output cap are two different limits
The context window is the total number of tokens a model can hold for one request, input and output together:
context window = input tokens + output tokens
On most providers the reasoning tokens are counted inside the output part. So if you set max_output_tokens too low, the reasoning can spend the whole allowance and leave little or no visible text. Plan headroom for it.
The output cap is a separate, smaller limit: the most the model will generate in one response. A model can accept a very large input and still cap its output far below that, and providers headline the first number rather than the second. As I write, the headline context windows on some models run to a million tokens, while the output caps sit well under that and are set per model. The numbers move often enough that I won't repeat them here. Check the page for your own deployment: Google Gemini tokens, OpenAI models, Anthropic context windows. Whatever those pages say today is an anchor for your config, not a guarantee.
Why you still budget at a million-token context window
An application that regularly exceeds its limits looks the same from the outside every time: slow responses, answers that stop mid-sentence, quality that gets worse as the conversation gets longer, and stakeholders who start to distrust "AI" in general and your team in particular. Four separate things go wrong, and each one is its own reason to budget.
Answers get cut off
A model doesn't know how long its answer will be before it writes it. Suppose Jenny in Legal asks a question whose full answer runs to 3,094 tokens, and the room left in the context window is 1,500. The model writes 1,500 tokens and stops. The other 1,594 never arrive. Because tokens are fragments of words, the answer stops mid-sentence, and Jenny sees a half-finished reply with no warning that anything went wrong. The interactive figure later in this essay lets you watch that happen.
So set an explicit output cap that matches what your application returns on each route, rather than leaving the provider default. And if you use a reasoning model, leave headroom above that cap, because the reasoning is charged against it, and a cap that's too tight returns an empty or partial answer with no error to tell you why (OpenAI developer community thread on the silent failure).
Quality drops as the input grows (context rot)
Context rot is the name for the drop in a model's ability to find and use information as the prompt gets longer. Newer models retrieve from long inputs better than older ones, but the drop hasn't gone away. You may have seen "needle in a haystack" results that suggest otherwise. That test plants one fact in a large body of text and asks the model to find it, and current models score well on it. The catch is that the needle usually shares vocabulary with the question, so the model does little reasoning and faces little ambiguity, which is not the shape of most real conversations or workflows.
At long input lengths, models still lose facts placed in the middle of a prompt and struggle to keep a long exchange coherent. The effect was first shown in Lost in the Middle (2023, arXiv:2307.03172). I'd treat it as a property of the approach rather than of any one model, and measure it on your own workload rather than trust either the paper or the marketing. The direct remedy is the same either way: keep the input to what the request needs.
Cost tracks tokens
Providers bill per token, input and output separately, and output is usually priced higher. So the cheapest request is the one that sends and asks for only what it needs, and if a route never needs a long answer, cap its output below the default.
Prompt caching helps on top of that. The provider stores the prefix of your prompt that repeats from call to call, the system prompt and the few-shot examples, and charges less when the next call starts with the same bytes. OpenAI documents a minimum prefix length of 1,024 tokens for a cache hit, with the cached prefix growing in 128-token steps (OpenAI prompt caching guide). Other providers set their own thresholds and lifetimes. The rule is the same in each case: put the stable material at the top of the prompt and keep it byte-identical between calls.
Latency and rate limits
More tokens means a slower response, and it moves you closer to the provider's tokens-per-minute (TPM) and requests-per-minute (RPM) limits. Cross those and your requests are throttled or dropped (Azure OpenAI quotas and limits). The caps you set for output length are the same caps that keep you under those limits, so the work is done once and pays twice: bounded latency, and fewer rate-limit errors.
Four ways to manage tokens
Here they are in order of effort. The first two are where most applications start. The third is where they should end up, and the rest of the essay is about doing it well.
Do nothing (fine for a throwaway prototype)
Set no caps, rely on the provider defaults, and ship. It's fast, and it fails in four predictable ways:
- Outputs truncate at random, whenever an answer happens to be long.
- Costs are unbounded on the worst-case input.
- The context fills as the interaction grows, for example over a long chat.
- Coherence and accuracy drop at the longer context lengths you drift into.
Use it for a prototype you'll throw away. Don't run it in production unless a hard spending cap sits in front of it.
Static caps (simple and predictable)
Pick a fixed limit per route: FAQ answers up to 512 tokens, drafted sections up to 2,048, and so on. Set it through the provider parameter, max_output_tokens in OpenAI's Responses API for example (OpenAI models). You get bounded cost and latency. What you don't get is any allowance for the outlier: a prompt that fills more of the window than you planned still truncates, because the cap never looked at how much room was left.
Dynamic budgeting (what I recommend)
Compute the output budget per request from four inputs: the model's context window, the provider's output cap, your own cap for the route (set from your cost and latency targets), and, on a reasoning model, the headroom you reserve for thinking. Subtract what the input already uses from the window and take the smallest of the three caps. The result is the most the model may write on this request without truncation.
This is the approach I'd choose every time, because it's the only one that spends exactly what the request needs. Counting tokens before every call adds some latency. In my experience it's small next to the cost of a truncated answer, but measure it on your own route before you decide.
See where the answer stops
The figure below turns the budget equation into the failure it prevents. It uses Jenny's 3,094-token answer from earlier inside an illustrative 32,000-token context window. The system prompt, her question and a reasoning reserve stay fixed. You move only the conversation history and watch where the answer stops.
How to use: drag conversation history. Watch the returned answer lose room at the orange boundary.
1,500 of 3,094 requested answer tokens fit. 1,594 tokens are not returned.
Hold 3,094 tokens before admitting history. The safe history limit here is 21,906.
Remove at least 1,594 history tokens to return the full answer.
Read it as the equation. Once the history plus the fixed pieces leaves fewer than 3,094 tokens, the answer is cut at exactly the shortfall, and the two remedies the figure offers are the two halves of dynamic budgeting: reserve the answer first, then trim the oldest turns until the rest fits.
The budget in code
The gist:
"""Compute a safe output-token budget from context and caps.
Args:
context_window: Total available tokens for the model (input + output + reasoning).
input_tokens: Tokens already used by the current request input.
reasoning_headroom: Reserved tokens for hidden reasoning on thinking models.
app_cap: Application-specific maximum allowed output tokens.
provider_output_cap: Model/provider maximum output tokens.
Returns:
max_output_tokens: Safe maximum output tokens for this request.
"""
max_out_for_context = context_window - input_tokens - reasoning_headroom
max_output_tokens = min(app_cap, provider_output_cap, max_out_for_context)The full implementation below adds the other half: it counts the tokens in the messages, summarises the oldest turns when they don't fit, and only then computes the cap. Read fit_messages_to_window first. Everything else supports it.
"""Utilities for dynamic token management with OpenAI Responses API and tiktoken.
This module counts tokens, trims/summarizes history to fit a context window,
and computes a safe max_output_tokens honoring application and provider caps.
It also reserves optional reasoning headroom for thinking models.
"""
# pip install openai tiktoken
import os
from typing import List, Dict, Any, Optional, Tuple
import tiktoken
from openai import OpenAI
import random
import time
MODEL_ID = os.getenv("OPENAI_MODEL", "gpt-5") # or your deployed model name
SUMMARIZER_MODEL = os.getenv("OPENAI_SUMMARIZER_MODEL", MODEL_ID)
CONTEXT_WINDOW = int(os.getenv("MODEL_CONTEXT_WINDOW", 128_000)) # set per-deployment
PROVIDER_OUTPUT_CAP = int(os.getenv("PROVIDER_OUTPUT_CAP", 8_192)) # see docs for your model
APP_OUTPUT_CAP = int(os.getenv("APP_OUTPUT_CAP", 2_048)) # your application's specific output cap
REASONING_HEADROOM = int(os.getenv("REASONING_HEADROOM", 0)) # e.g., 1024+ for reasoning models
MIN_OUTPUT_FLOOR = int(os.getenv("MIN_OUTPUT_FLOOR", 256))
MAX_SUMMARY_PASSES = int(os.getenv("MAX_SUMMARY_PASSES", 3))
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
def _enc_for_model(model: str):
"""Return a tokenizer encoding for the given model, with sensible fallback.
Args:
model: Model name hint used to resolve the most appropriate tokenizer.
Returns:
A tiktoken ``Encoding`` instance suitable for the provided model.
Notes:
Falls back to the generic ``cl100k_base`` encoding when the model is
unknown to ``tiktoken.encoding_for_model``.
"""
try:
return tiktoken.encoding_for_model(model)
except Exception:
return tiktoken.get_encoding("cl100k_base")
def count_text_tokens(text: str, model: str) -> int:
"""Count tokens for a text string using the model's tokenizer.
Args:
text: The input string to tokenize.
model: Model name used to choose the correct encoding.
Returns:
The number of tokens produced by encoding ``text``.
"""
enc = _enc_for_model(model)
return len(enc.encode(text))
def count_message_tokens(messages: List[Dict[str, Any]], model: str) -> int:
"""Estimate token count for chat messages, including envelope overhead.
Args:
messages: A list of message dicts with a ``content`` field. ``content``
may be a string or a list of parts (strings or dicts with ``text``
/ ``input`` fields).
model: Model name used to choose the correct encoding.
Returns:
Estimated total token count for all messages.
Notes:
Adds a 4‑token per-message overhead and a 2‑token envelope overhead to
better approximate OpenAI-style chat tokenization.
"""
total = 0
for m in messages:
c = m.get("content", "")
if isinstance(c, list):
flat = "".join(
str(p.get("text") or p.get("input") or p)
if isinstance(p, dict) else str(p)
for p in c
)
total += count_text_tokens(flat, model)
else:
total += count_text_tokens(str(c), model)
total += 4 # heuristic per-message overhead
total += 2 # envelope overhead
return total
def summarize_block(block: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Summarize a block of messages into a compact, reusable system note.
Args:
block: Older messages to condense while preserving salient facts.
Returns:
A single system message dict containing the concise summary.
Notes:
Uses the currently configured model at low temperature and caps the
summary to 256 output tokens to control growth.
"""
summary_input = [
{"role": "system", "content": "Summarize concisely; keep only reusable facts."},
{"role": "user", "content": "\n\n".join(f"{m['role'].upper()}: {m.get('content','')}" for m in block)},
]
resp = client.responses.create(
model=SUMMARIZER_MODEL,
input=summary_input,
max_output_tokens=256,
temperature=0
)
return {"role": "system", "content": "[EARLIER SUMMARY]\n" + resp.output_text}
def fit_messages_to_window(msgs: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Trim/summarize messages so there is safe room for model output.
Ensures the remaining context allows at least ``MIN_OUTPUT_FLOOR`` tokens
of output after reserving ``REASONING_HEADROOM``.
Args:
msgs: Original list of chat messages.
Returns:
A possibly condensed list of messages that fits the context window.
Notes:
Prefers summarizing older turns first; as a last resort, truncates the
longest recent user input if still over budget.
"""
messages = list(msgs)
def fits(ms):
used = count_message_tokens(ms, MODEL_ID)
return (CONTEXT_WINDOW - used - REASONING_HEADROOM) >= MIN_OUTPUT_FLOOR
# quick exit
if fits(messages):
return messages
# summarize older turns first
older, recent = (messages[:-2], messages[-2:]) if len(messages) > 3 else ([], messages)
passes = 0
while older and passes < MAX_SUMMARY_PASSES:
mid = max(1, len(older)//2)
older = [summarize_block(older[:mid])] + older[mid:]
if fits(older + recent):
return older + recent
passes += 1
# last resort: trim long recent input
while not fits(older + recent):
i = max(range(len(recent)), key=lambda k: len(str(recent[k].get("content",""))))
c = str(recent[i].get("content",""))
if len(c) < 400: break
recent[i]["content"] = c[:300] + "\n[...truncated to fit context...]"
return older + recent
def compute_max_output_tokens(messages: List[Dict[str, Any]]) -> int:
"""Compute a safe ``max_output_tokens`` within context and cap limits.
Args:
messages: Messages to be sent to the model (already fitted if needed).
Returns:
The maximum number of output tokens permitted for this request.
Raises:
ValueError: If the computed budget falls below ``MIN_OUTPUT_FLOOR``.
"""
used = count_message_tokens(messages, MODEL_ID)
context_room = max(0, CONTEXT_WINDOW - used - REASONING_HEADROOM)
provisional = min(APP_OUTPUT_CAP, PROVIDER_OUTPUT_CAP, context_room)
if provisional < MIN_OUTPUT_FLOOR:
raise ValueError("Insufficient room for a safe output; reduce input further.")
return provisional
def generate(messages: List[Dict[str, Any]], stop: Optional[List[str]] = None, temperature: float = 0.2):
"""Generate a response with a dynamically budgeted output token cap.
Args:
messages: Chat messages to send to the model.
stop: Optional list of stop strings to end generation early.
temperature: Sampling temperature for the model.
Returns:
A dict with ``text`` (model output) and ``usage`` (token accounting).
Raises:
ValueError: If there is insufficient room for minimum safe output.
"""
fitted = fit_messages_to_window(messages)
max_out = compute_max_output_tokens(fitted)
resp = client.responses.create(
model=MODEL_ID,
input=fitted,
max_output_tokens=max_out,
temperature=temperature,
stop=stop or []
# optionally: add structured output schema here
)
usage = getattr(resp, "usage", None)
reasoning = None
if usage and getattr(usage, "completion_tokens_details", None):
reasoning = usage.completion_tokens_details.reasoning_tokens
return {
"text": resp.output_text,
"usage": {
"input_tokens": getattr(usage, "input_tokens", None),
"output_tokens": getattr(usage, "output_tokens", None),
"reasoning_tokens": reasoning
}
}The defaults in that code are conservative on purpose. Confirm your model's context window and output cap from the provider docs before you rely on them (OpenAI models, Google Gemini tokens).
Once the arithmetic is in place, the next question is whether it's working, and only telemetry can answer that.
Measure whether it's working
Log these per route:
- p50 and p95 input tokens, and the same for output tokens
- truncation rate: the share of responses that hit the cap
- cache hit rate, if you use prompt caching
- TPM and RPM headroom, and how often you backed off
- cost per request and cost per route
An illustrative log line:
{"route":"faq","p50_in":480,"p95_in":1900,"p50_out":220,"p95_out":800,"truncated":0.07,"cache_hit":0.61,"tpm_headroom":0.32,"rpm_headroom":0.45,"cost_cents":1.8}Put these on one dashboard and alert on two of them: the truncation rate rising, and the rate-limit headroom falling. Those are the two that users feel first.
Degrade when the budget is tight
The numbers will move as prompts, users and models change. When they do, change what the application returns rather than let the request fail:
- Return a short structured form (a summary and three bullets) when the computed budget falls below a floor you set, 400 tokens say.
- Ask the user back when the input alone blows the budget: "This is too long. Summarise the source, or narrow the question?"
- Support deliberate chunking: end with a clear "continue" signal and resume from it on the next call.
A structured output starter:
{"summary":"","bullets":["","",""],"sources":[]}Prefer structured outputs generally. They bound the length of the answer, and they make parsing and evaluation straightforward. If you use tool calls or JSON mode, reserve extra output headroom for the serialisation overhead, and add stop sequences so the model can't run on past the end of the structure.
Cheat sheet
- Set a floor (
min_output_floor, 256 in the code above) so you never return a schema-invalid half-answer. - On a reasoning model, reserve headroom for thinking. My own rule of thumb is 15 to 25 percent of the output budget. I haven't measured that across models, so treat it as a starting point and tune it from your reasoning-token telemetry.
- Keep the cacheable prefix byte-identical and at the top of the prompt. Minimum lengths and lifetimes vary by provider.
- Set explicit caps per route. A default is not a product decision.
- Summarise and prune chat history before it crowds out the answer. Long prompts multiply both latency and cost.
- Monitor p95 input and output tokens and the truncation rate. When the budget is tight, degrade to the structured short form.
What to do now
Tokens are a resource, and you budget them the way you already budget CPU time and memory. Per request: compute the output budget from the context window, the provider cap, your route cap and the reasoning headroom. Cap output to what the route needs. Prune and summarise history before it crowds out the answer. Prefer structured outputs. Keep the cacheable prefix stable. Measure input, output and truncation, and alert on them.
Context windows will keep growing and the price per token will keep changing, but none of those steps changes with them. What does change is the model card your config depends on, and it changes more often than your code does. Put a date on the limits in your config and check them against the provider docs every time you change models.
The decision that's left is what your application should return when the budget is short. That's a product decision, and if you haven't made it, the provider defaults have already made it for you.