Preventing Slow and Expensive LLM Responses When Prompt Context Keeps Growing

A team ships an LLM-backed support assistant. The first version is simple: take the user's question, append a large pile of documentation, add the last few conversation messages, and send everything to the model. It works in a demo. Then production traffic arrives.

The symptoms are familiar. Some requests take too long because the model must read a huge prompt and generate a long answer. Token bills grow because every request repeats the same background text. The assistant still misses important facts because the relevant paragraph was not included, or because it retrieved similar but unhelpful content. Tests are flaky because the same prompt does not always produce the same wording.

The fix is not a better magic prompt. The fix is a context pipeline. Instead of dumping data into the model, the system should decide what information belongs in the model's short-term memory for this exact request. That means budgeting tokens, cleaning source data, retrieving relevant chunks, adding relationship-aware lookup when similarity is not enough, and testing the result with metrics that match LLM behaviour.

This post walks through a practical design for that pipeline using a support assistant as the running example. The same pattern applies to code assistants, search assistants, learning platforms, and internal knowledge bots.

The Problem

A large language model, or LLM, generates text by working with tokens. A token is the unit of text the model processes. It may be a word, part of a word, punctuation, or another small text fragment depending on the tokenizer.

Tokens matter for three reasons:

• They define cost, because LLM APIs are commonly billed by token volume.
• They define latency, because output tokens are generated sequentially.
• They define memory, because every model has a maximum context window.

The context window is the amount of text the model can see at one time. It includes system instructions, user messages, conversation history, retrieved documents, tool outputs, and the model's response. If the context window is filled with low-value content, the model may ignore or miss the key facts.

For a support assistant, the inputs usually look like this:

• The current user question.
• Recent conversation history.
• Product documentation.
• Troubleshooting articles.
• Closed support tickets.
• Account or tenant restrictions.
• System instructions and safety rules.

The expected output is a grounded answer that uses only relevant facts, avoids inventing information, and responds fast enough that the user does not feel blocked.

The dangerous version of the architecture looks like this:

User question
  |
  v
Append recent chat history
  |
  v
Append many documents
  |
  v
Send one huge prompt to LLM
  |
  v
Hope the answer is correct

This design fails because it treats the model like an infinite memory store. It is not. The model is a probabilistic compute engine with a strict working memory limit.

What the System Needs to Do

A production context pipeline should do five jobs before calling the model:

1. Remove unsafe or low-value source text.
2. Convert searchable knowledge into embeddings.
3. Retrieve the best candidate chunks for the current question.
4. Fit the final context into a token budget.
5. Measure quality, latency, and failure modes after every change.

An embedding is a vector representation of text. Similar meanings should end up close to each other in vector space. This lets the system find documentation that is conceptually related to the user's question, even when the exact words do not match.

Retrieval-augmented generation, usually called RAG, uses that idea in a workflow: retrieve relevant facts, add them to the prompt, then ask the model to generate the final response from that context.

A better support assistant architecture looks like this:

User question
  |
  v
Normalize and classify request
  |
  v
Retrieve related chunks from vector index
  |
  v
Optionally traverse knowledge graph relationships
  |
  v
Rank, deduplicate, and compress context
  |
  v
Assemble prompt within token budget
  |
  v
Call LLM with task-appropriate settings
  |
  v
Validate, log metrics, and return response

The rest of the post turns this into a practical workflow.

Step 1: Treat the Context Window Like a Budget

Start by dividing the context window into explicit buckets. Do not let retrieved documents consume everything.

A simple budget might look like this:

The exact numbers depend on the model, but the structure matters more than the values. Every request should reserve space for the answer. Otherwise, the system can build a prompt so large that the model has little room left to respond.

Here is a conceptual Python helper that shows the idea without tying it to a specific tokenizer or vendor API:

from dataclasses import dataclass

@dataclass
class ContextItem:
    label: str
    text: str
    priority: int


def rough_token_count(text: str) -> int:
    words = text.split()
    return max(1, int(len(words) * 1.4))


def choose_context_items(items: list[ContextItem], max_tokens: int) -> list[ContextItem]:
    selected: list[ContextItem] = []
    used_tokens = 0

    for item in sorted(items, key=lambda current: current.priority, reverse=True):
        item_tokens = rough_token_count(item.text)
        if used_tokens + item_tokens <= max_tokens:
            selected.append(item)
            used_tokens += item_tokens

    return selected

This is not a replacement for a real tokenizer. It is a design guardrail. The important rule is that context assembly must be intentional and measurable.

For conversation memory, use one of two strategies:

• Sliding window: keep the latest turns and drop older ones when the window fills.
• Summarization: compress older turns into a short memory note, then keep the latest turns verbatim.

Sliding windows are simple and preserve recent detail. Summaries preserve long-term continuity but can lose nuance. Many systems use both.

Step 2: Clean Knowledge Before Embedding It

Bad source text produces bad retrieval. Before creating embeddings, clean the data that will enter the retrieval index.

For support documentation, cleaning usually includes:

• De-noising: remove headers, footers, navigation labels, and repeated boilerplate.
• Sensitive data scrubbing: remove emails, tokens, account identifiers, and secrets before sending text to an external model or embedding service.
• Normalization: standardize casing, whitespace, date formats, and unusual characters.

A small cleaning stage can prevent a lot of downstream problems:

import re

SECRET_PATTERNS = [
    re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}"),
    re.compile(r"\b(?:api|secret|token)[_-]?[A-Za-z0-9]{12,}\b", re.IGNORECASE),
]


def clean_document(raw_text: str) -> str:
    text = raw_text.replace("\r\n", "\n")
    text = re.sub(r"\n{3,}", "\n\n", text)
    text = re.sub(r"(?i)footer:.*", "", text)

    for pattern in SECRET_PATTERNS:
        text = pattern.sub("[REMOVED]", text)

    return text.strip()

This example is intentionally small. In a real system, the cleaning rules should be reviewed by the teams responsible for security, privacy, and compliance. The architectural point is simple: do not wait until prompt assembly to remove risky or useless data.

Step 3: Retrieve by Meaning, Not by Exact Words

Keyword lookup works when users use the same terms as the documentation. Support users rarely do that. One user writes "reset password" while another writes "cannot sign in after changing phone." Both may need the same authentication article.

Vector search solves part of that problem:

1. Split cleaned documents into meaningful chunks.
2. Convert each chunk into an embedding vector.
3. Store the vector and chunk metadata in a vector database.
4. Convert the user's question into an embedding.
5. Search for nearby vectors using a similarity measure such as cosine similarity.
6. Pass the best matching chunks to the model as context.

Conceptually, the retrieval function looks like this:

from dataclasses import dataclass

@dataclass
class RetrievedChunk:
    source_id: str
    text: str
    score: float


def retrieve_context(question: str, tenant_id: str, vector_store) -> list[RetrievedChunk]:
    question_vector = embed_text(question)

    matches = vector_store.search(
        vector=question_vector,
        limit=12,
        filter_by_tenant=tenant_id,
    )

    useful_matches = []
    for match in matches:
        if match.score >= 0.78:
            useful_matches.append(
                RetrievedChunk(
                    source_id=match.source_id,
                    text=match.text,
                    score=match.score,
                )
            )

    return useful_matches

The tenant filter is not optional in a multi-tenant support system. Retrieval must not search across all customers and rely on the model to ignore the wrong data. The retrieval layer should enforce access boundaries before the prompt exists.

Step 4: Know When Naive RAG Is Not Enough

Vector search finds semantic similarity. That is useful, but similarity is not always the same as relevance.

Two common failures appear in support assistants:

• Redundant retrieval: several chunks say nearly the same thing, so the model receives repeated context instead of broader evidence.
• Hierarchy mismatch: a query matches a document title, but the actual fix is buried several sections below.

This is where a knowledge graph can help. A knowledge graph stores entities as nodes and relationships as edges. For support data, nodes might represent documents, sections, incidents, known errors, releases, commits, and product features.

A graph can answer relationship questions that vector similarity may miss:

TroubleshootingGuide
  has_section -> LoginFailures
LoginFailures
  mentions_error -> SessionExpired
SessionExpired
  resolved_by -> PatchRelease_2025_04
PatchRelease_2025_04
  changes_component -> AuthGateway

GraphRAG combines vector retrieval with graph traversal. The vector index finds likely entry points. The graph follows relationships to collect the connected facts that make the answer complete.

Use graph traversal when questions depend on structure, ownership, dependency, or cause and effect. Use vector search when questions depend mainly on semantic similarity.

Step 5: Assemble the Prompt as a Controlled Data Product

Prompt engineering is useful, but context engineering is the larger design problem. The system must construct the model's information environment on every request.

For the support assistant, the final prompt assembly should include:

• Stable system instructions that define the assistant's role and boundaries.
• The current user question.
• A compact summary of relevant conversation state.
• Ranked retrieved facts with source labels.
• Any required constraints, such as answer format or forbidden behavior.

The context builder should avoid sending every retrieved chunk directly. Rank, deduplicate, and compress first.

def build_prompt(question: str, memory: str, chunks: list[RetrievedChunk]) -> str:
    ranked_chunks = sorted(chunks, key=lambda chunk: chunk.score, reverse=True)
    evidence_blocks = []

    for index, chunk in enumerate(ranked_chunks[:6], start=1):
        evidence_blocks.append(
            f"Evidence {index} from {chunk.source_id}:\n{chunk.text}"
        )

    return "\n\n".join(
        [
            "You are a support assistant. Use only the evidence provided.",
            "If the evidence is insufficient, say what is missing.",
            f"Conversation memory:\n{memory}",
            f"User question:\n{question}",
            "Evidence:\n" + "\n\n".join(evidence_blocks),
        ]
    )

This keeps the model grounded. It also makes the request easier to debug because the retrieved evidence is explicit.

Step 6: Configure Variance Based on the Task

LLM outputs are probabilistic. The same input can produce different wording, and sometimes different structure, unless you constrain generation.

Temperature is the main control knob. A low temperature makes the model prefer more likely next tokens, which is better for classification, extraction, code generation, and support answers. A higher temperature increases variation, which is more useful for brainstorming or creative generation.

For a support assistant, prefer low temperature. The goal is not novelty. The goal is repeatable, grounded help.

This also affects retries. If a low-temperature request fails validation, retrying the exact same request may fail in the same way. A better fallback is to change the prompt, reduce the context, use another model, or return a graceful message that the system needs more information.

Step 7: Benchmark Quality and Speed Separately

Do not judge an LLM feature only by whether a few manual tests feel good. Measure it along two dimensions.

Quality benchmarking asks: did the assistant answer correctly?

Use a golden set: a small collection of representative questions, expected answer criteria, and source documents. Score each run for accuracy, groundedness, and usefulness. Human review is the strongest option. A stronger model can also be used as a judge when manual review is too slow, but the scoring rules must be clear.

Inference benchmarking asks: did the assistant respond fast enough and cheaply enough?

Track these metrics:

• Time to first token: how long until the user sees the first generated text.
• Tokens per second: generation speed after the first token appears.
• Total latency: full time from request to completed answer.
• Throughput: requests handled per second.
• Resource usage: memory and compute needed to serve the model.
• Token cost per request: input and output token volume.

Quality and speed can move in opposite directions. Adding more retrieved context may improve correctness but increase cost and latency. A smaller model may respond faster but miss harder reasoning tasks. The architecture should make those trade-offs visible.

Handling Common Failure Modes

Hallucination

A hallucination is a confident but false answer. Reduce it by grounding the answer in retrieved facts and instructing the model to admit when evidence is missing.

Stale knowledge

A model's trained knowledge can be out of date. RAG helps by supplying current documentation, tickets, and database facts at request time.

Bias and unsafe behaviour

Some bias comes from training data. Manage it with data filtering, prompt guardrails, human feedback, and output validation.

Context overflow

When the final request exceeds the model's context window, do not blindly truncate from the end. Use priorities. Keep the user question, safety instructions, and highest-value evidence. Summarize or drop lower-priority history.

Common Mistakes

• Caching only final answers while recomputing embeddings for the same text repeatedly.
• Sending entire documents when a few chunks would answer the question.
• Using vector similarity without tenant or user access filters.
• Treating low similarity as proof that no answer exists.
• Forgetting that output tokens affect latency more visibly than input tokens.
• Testing responses with exact string comparisons instead of quality criteria.
• Choosing a large model for every task instead of routing simple tasks to cheaper or faster models.
• Assuming embeddings are a privacy guarantee. They reduce direct readability, but they are not the same as encryption.

Checklist

Use this checklist before moving an LLM support assistant from demo to production:

• Define the maximum token budget per request.
• Reserve output space before adding retrieved context.
• Clean and scrub data before embedding.
• Chunk documents by meaning, not arbitrary character count.
• Store embeddings with access-control metadata.
• Use vector search for semantic retrieval.
• Add graph traversal when relationships matter.
• Rank and deduplicate retrieved chunks.
• Use summaries or sliding windows for long conversations.
• Set low temperature for support, extraction, and classification tasks.
• Track time to first token, total latency, token cost, and quality scores.
• Test with a golden set instead of exact output strings.
• Provide a graceful fallback when the system lacks evidence.

Conclusion

Growing prompt context is not just a prompt-writing problem. It is a system design problem. Tokens affect cost, latency, and memory. Embeddings help retrieve relevant facts, but naive RAG can still miss relationship-heavy answers. Knowledge graphs help when the system must follow structure instead of only similarity.

A reliable LLM application treats context as a controlled pipeline: clean the data, retrieve the right facts, fit them into a budget, configure generation for the task, and measure the result. That is how you keep responses useful without letting prompts become slow, expensive, and unreliable.