Building an LLM Gateway to Keep Slow, Costly Model Calls from Spreading Across Microservices

A team usually starts with the fastest possible integration: one service imports a model provider SDK, another service uses a different provider, and a third service stores a separate API key. That works while the feature is a prototype. It becomes painful when the product has real users.

The first symptoms are not always dramatic outages. They often look like small inconsistencies:

• One endpoint retries for too long while a user waits on a spinner.
• Another endpoint fails immediately on a temporary provider timeout.
• Different services use different prompt formats and safety rules.
• API keys are copied into too many deployment environments.
• Cost reports are split across several provider dashboards.
• A model upgrade improves one workflow but breaks another.

The fix is not to ask every product team to become expert at LLM operations. The safer design is to put an internal LLM gateway between product services and model providers. Product services call one internal API. The gateway handles routing, retries, fallbacks, streaming, caching, retrieval, cost controls, observability, and security.

This post walks through a practical gateway design for a multi-service application. The goal is not to hide every detail of model usage. The goal is to centralize the details that must be consistent in production.

The Problem

A large language model, or LLM, is not like a normal deterministic helper function. The same input can produce slightly different output, latency is often much higher than a database lookup, and the cost of a request depends on the amount of input and output text processed by the model.

That creates a new dependency profile for production systems:

Direct model calls spread these concerns everywhere. The product code becomes coupled to provider SDKs, each team invents its own retry behavior, and no single service can answer basic operational questions such as, "Which model is causing our cost spike?" or "Which users are hitting the budget ceiling?"

What the Gateway Owns

The gateway is an internal service that receives normalized LLM work requests from the rest of the platform. It is the only service allowed to call external or self-hosted models.

A simple deployment shape looks like this:

Product Service A
Product Service B
Product Service C
        |
        v
Internal LLM Gateway
        |
        +--> Model router
        +--> Prompt builder
        +--> Cache layer
        +--> Circuit breakers
        +--> Retrieval layer
        +--> Safety filters
        +--> Metrics and cost tracker
        |
        v
Provider 1 / Provider 2 / Local model

The gateway should own these responsibilities:

1. Provider abstraction: product services should not know whether the final call goes to a premium reasoning model, a fast smaller model, or a locally hosted fallback.
2. Central authentication: provider keys should live in one service, not in every application service.
3. Routing and cost control: simple tasks should not use the most expensive model by default.
4. Resilience: timeouts, retries, circuit breakers, and fallbacks should be consistent.
5. Latency strategy: synchronous, asynchronous, streaming, and cached flows should be chosen deliberately.
6. Grounding: retrieval-augmented generation, or RAG, should supply trusted context when the model needs private or current data.
7. Safety: prompt injection checks, output validation, tenant filters, and rate limits should be enforced before risky output reaches the application.
8. Evaluation and observability: quality, cost, latency, and provider health should be measured centrally.

Define One Internal Request Shape

The gateway needs enough information to route and protect a request without exposing provider-specific details to callers.

from dataclasses import dataclass
from enum import Enum

class TaskKind(Enum):
    GRAMMAR_FIX = "grammar_fix"
    CHAT_REPLY = "chat_reply"
    CODE_HELP = "code_help"
    REPORT_BUILD = "report_build"
    CLASSIFICATION = "classification"

@dataclass
class GatewayCall:
    request_id: str
    tenant_id: str
    user_id: str
    task: TaskKind
    user_plan: str
    prompt_text: str
    estimated_input_tokens: int
    max_output_tokens: int
    needs_private_context: bool
    interactive: bool

@dataclass
class GatewayResult:
    text: str
    model_name: str
    cache_hit: bool
    fallback_used: bool
    estimated_cost_units: float

The exact transport can be HTTP, gRPC, or an internal RPC mechanism. The important part is the contract. A caller asks for a business task. It does not choose a provider SDK, implement retries, or decide how to handle model overload.

Route to the Cheapest Good Enough Model

One of the most expensive mistakes in LLM systems is using the strongest model for every task. A simple classification, grammar correction, or formatting task often does not need the same model as a complex reasoning workflow.

A model router is a rules layer inside the gateway. It can consider task type, user tier, model health, token count, latency, and budget.

class ModelChoice:
    def __init__(self, name, reason):
        self.name = name
        self.reason = reason

class ProviderHealth:
    def __init__(self, p99_latency_ms, error_rate, rate_limited):
        self.p99_latency_ms = p99_latency_ms
        self.error_rate = error_rate
        self.rate_limited = rate_limited


def choose_model(call, health_by_model, daily_spend_units):
    if call.estimated_input_tokens > 8000:
        return ModelChoice("budget_long_context_model", "large input routed away from premium model")

    if daily_spend_units > 10:
        return ModelChoice("local_small_model", "user reached soft daily cost limit")

    if call.task == TaskKind.GRAMMAR_FIX:
        return ModelChoice("local_small_model", "simple language cleanup")

    if call.task == TaskKind.CLASSIFICATION:
        return ModelChoice("fast_classifier_model", "short deterministic decision")

    premium = health_by_model["premium_reasoning_model"]
    if call.user_plan == "premium" and not premium.rate_limited and premium.p99_latency_ms <= 2000:
        return ModelChoice("premium_reasoning_model", "complex task for premium user")

    return ModelChoice("fast_general_model", "default healthy model")

This is not only a cost optimization. It is also a reliability tool. When the premium provider is congested and its P99 latency breaches the service target, the router can temporarily shift traffic to a faster model. That is better than letting every request pile up behind a slow dependency.

Use Circuit Breakers Before Retries Make Things Worse

Blind retries are dangerous with LLM providers. If the provider is already slow or returning errors, aggressive retries can increase load and make user-facing latency worse.

A circuit breaker watches provider health. When failures or latency cross a threshold, it opens the circuit and stops sending normal traffic to that provider. After a cooldown, it allows a small probe request. If the probe succeeds, the circuit closes. If it fails, the cooldown starts again.

import time

class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"

class CircuitBreaker:
    def __init__(self, cooldown_seconds):
        self.state = CircuitState.CLOSED
        self.cooldown_seconds = cooldown_seconds
        self.opened_at = 0

    def allow_request(self):
        if self.state == CircuitState.CLOSED:
            return True

        if self.state == CircuitState.OPEN:
            elapsed = time.time() - self.opened_at
            if elapsed >= self.cooldown_seconds:
                self.state = CircuitState.HALF_OPEN
                return True
            return False

        if self.state == CircuitState.HALF_OPEN:
            return False

    def record_success(self):
        self.state = CircuitState.CLOSED

    def record_failure(self):
        self.state = CircuitState.OPEN
        self.opened_at = time.time()

Pair the breaker with tiered fallbacks:

Retry strategy should depend on whether a user is waiting:

• Interactive path: use a short capped retry, then fail fast to a fallback. A long backoff feels like downtime to the user.
• Asynchronous path: use longer exponential backoff because the user is not blocked on the request.

Split Synchronous and Asynchronous Work

Not every LLM task belongs on the same request path.

A code completion, search rewrite, or short classification may need a fast synchronous response. A long report, batch content generation job, or agent workflow should not block an HTTP request while the model works.

User request
    |
    v
API Gateway
    |
    +--> Synchronous path
    |       |
    |       +--> Fast model or cache
    |       +--> Immediate response
    |
    +--> Asynchronous path
            |
            +--> Queue
            +--> Worker
            +--> Reasoning model
            +--> Result store
            +--> Polling or callback

A useful rule is to keep low-latency work on the synchronous path and send long-running work to a queue. The caller can receive an accepted response immediately, then poll for completion or receive a push notification when the result is ready.

For chat and conversational interfaces, streaming improves perceived latency. The model may still take several seconds to finish, but the user sees the first tokens quickly.

def stream_tokens(model_client, prompt_text):
    for token in model_client.generate_stream(prompt_text):
        yield "data: " + token + "\n\n"


def response_headers():
    return [
        ("content-type", "text/event-stream"),
        ("cache-control", "no-cache"),
    ]

Server-sent events work well for one-way streaming from server to browser because they run over normal HTTP and are simple to consume from a web client.

Cache at More Than One Level

Caching is one of the fastest ways to reduce both latency and cost. A cache hit is the cheapest model call because it avoids the model call entirely.

Use multiple cache levels:

Exact-match caching can hash a normalized prompt. Semantic caching stores an embedding of the prompt and checks whether a similar request already has a safe cached answer. Proactive caching moves work out of the online path. Coalescing prevents a cache stampede during traffic spikes.

import asyncio

class CoalescingCache:
    def __init__(self):
        self.values = dict()
        self.pending = dict()

    async def get_or_create(self, key, producer):
        if key in self.values:
            return self.values[key], True

        if key in self.pending:
            return await self.pending[key], True

        task = asyncio.create_task(producer())
        self.pending[key] = task
        try:
            value = await task
            self.values[key] = value
            return value, False
        finally:
            self.pending.pop(key, None)

Cache invalidation matters. A cached answer may need to be invalidated when source content changes, a prompt template changes, a model version changes, or a safety rule changes.

Ground Answers with RAG Instead of Hoping the Model Knows

RAG means retrieval-augmented generation. The system retrieves trusted context first, adds that context to the prompt, and asks the model to answer using only that context.

This solves two common issues:

• The model may not know private company data.
• The model may have stale knowledge because its training data has a cutoff.

A basic RAG flow looks like this:

User question
    |
    v
Retrieve relevant context
    |
    +--> Database lookup for exact business facts
    +--> Vector search for semantically related documents
    +--> Graph search for connected entities when relationships matter
    |
    v
Build grounded prompt
    |
    v
Generate answer

The retrieval layer needs an ingestion pipeline. Documents are chunked into meaningful pieces, each chunk is embedded through an embedding model, and the chunk plus vector is stored in a vector database. When a user asks a question, the query is embedded and compared with stored vectors to find related chunks.

Vector search is strong for semantic similarity, but it can struggle when the answer depends on structured relationships. For example, similar documents are not always the most relevant documents. A hybrid RAG design can combine vector search with a knowledge graph, so the system can retrieve both semantically similar content and relationship-based context.

The gateway is a good place to coordinate this because it already owns the prompt construction step.

Treat Model Output as Untrusted

An LLM response should not be treated as safe just because it came from your own backend. It can contain wrong facts, unsafe instructions, malformed structured output, leaked prompt text, or content influenced by prompt injection.

Use layered controls:

1. Separate instructions from user data. Keep trusted system instructions separate from untrusted user input.
2. Filter suspicious input. A lightweight model or rule layer can reject likely prompt injection before it reaches the expensive model.
3. Validate output. If the application expects a structured response, parse it and validate the expected fields before using it.
4. Never execute raw model output. Do not evaluate code, run shell commands, or execute database statements generated directly by a model.
5. Use plan-approve-execute for agents. The model can propose an action plan, but application code should approve permissions before execution.
6. Require human approval for high-impact actions. Refunds, deletions, permission changes, and destructive operations should not be fully autonomous.

For retrieval, tenant filtering is critical. The retrieval layer must filter by tenant or user before context reaches the model. Do not search across all tenants and trust the model to ignore the wrong data.

def retrieve_context(call, vector_store):
    filters = dict()
    filters["tenant_id"] = call.tenant_id
    filters["user_id"] = call.user_id

    if not call.needs_private_context:
        return []

    return vector_store.search(
        text = call.prompt_text,
        filters = filters,
        limit = 8,
    )

Also defend against model denial of service. A user can overload the system with very long prompts or repeated expensive tasks. Enforce per-user request limits, reject abusive inputs, and downgrade users to cheaper models when they exceed a soft budget.

Measure What Normal Dashboards Miss

Standard service metrics are still useful, but they are not enough. CPU, memory, and error rates do not tell you whether the model became too expensive, too slow to stream, or worse at answering.

Add LLM-specific metrics:

Testing also has to change. Exact string assertions are fragile because model output can vary. Use golden datasets: representative inputs with ideal answers or expected quality criteria. Then run the system against those examples in CI/CD.

For scalable evaluation, use an evaluator model as a judge. It receives the original user input, the expected answer or rubric, and the actual answer, then assigns scores for accuracy, groundedness, and tone. For code workflows, deterministic checks such as compiler or interpreter validation can catch syntax failures without using another model.

def evaluate_answer(case, actual_text, judge):
    score = judge.score(
        user_task = case.user_task,
        ideal_answer = case.ideal_answer,
        actual_answer = actual_text,
        criteria = ["accuracy", "grounding", "tone"],
    )

    if score.overall < case.minimum_score:
        return "block_release"

    return "allow_release"

The exact scoring mechanism can vary, but the release gate should be explicit. A model or prompt change should not ship if it drops below the quality threshold for important tasks.

Common Mistakes

Letting every service call providers directly

This creates repeated retry code, scattered credentials, inconsistent logs, and painful provider migrations. Put provider calls behind one gateway.

Retrying interactive requests for too long

A long retry loop can look like downtime to a user. Use short capped retries for interactive traffic and move long retries to asynchronous workers.

Routing every task to the strongest model

A premium reasoning model is often unnecessary for classification, grammar cleanup, or formatting. Route based on task complexity and user value.

Caching without safety boundaries

Semantic cache hits are powerful, but cached answers must be safe to reuse. Include tenant, model version, prompt version, and data freshness in cache decisions.

Running RAG without tenant filters

Retrieval must be scoped before context reaches the model. Tenant-level filtering is an architectural control, not a prompt instruction.

Trusting model output because it came from your backend

Model output is still untrusted. Validate, sanitize, and restrict what it can trigger.

Production Checklist

Use this checklist before allowing product teams to depend on the gateway:

• Product services call only the internal gateway for LLM work.
• Provider credentials are centralized and not copied into every service.
• The router can select models by task, cost, health, token count, and user tier.
• Circuit breakers track latency and failures per provider.
• Interactive traffic has short retries and fallbacks.
• Long-running work uses queues and worker retries.
• Streaming is available for chat-style responses.
• Exact, semantic, proactive, and coalesced caching are considered where safe.
• RAG retrieval is scoped by tenant or user before prompt construction.
• Prompt injection, unsafe output, excessive agency, and sensitive data exposure have explicit controls.
• Rate limits and cost throttles protect against model denial of service.
• LLM-specific metrics are visible in dashboards.
• Golden datasets and evaluator checks run before prompt or model changes ship.

Conclusion

The gateway pattern turns LLM usage from scattered SDK calls into an internal platform capability. It gives product services a stable interface while centralizing the hard parts: routing, fallbacks, latency handling, caching, grounding, security, cost control, and quality evaluation.

The biggest benefit is operational consistency. When a provider slows down, the gateway can reroute. When costs spike, the gateway can throttle or downgrade. When a model update changes output quality, the evaluation pipeline can catch it before users do. That is the difference between adding an LLM feature and operating an LLM-backed system.