Preventing Slow E-Commerce Search When LLM Query Rewrites Move Into the Request Path

A product search box looks simple until users stop typing exact product names. Someone searches for "snacks for a movie night", "comfortable shoes for my grandfather", or "light jacket for a cold hill trip". A keyword index can match exact terms, synonyms, and filters, but it often misses the intent behind the query.

The tempting fix is to call a large language model (LLM) on every search request. The model can rewrite vague queries, infer categories, and suggest related products. That works in a demo, but it is dangerous in production. LLM calls are slower than normal service calls, cost grows with traffic, and provider failures can turn search into an outage.

For a mid-to-large e-commerce platform, the numbers make the problem clear. With 10 million daily active users and 5 searches per user per day, the platform handles about 50 million searches daily. That is about 579 average requests per second, and about 2,895 requests per second at 5x peak. The search experience should usually respond in less than a second at P90, while difficult P99 searches can tolerate a few seconds only when that extra time avoids a dead-end result.

The goal is not to remove LLMs from search. The goal is to put them in the right place: mostly offline, sometimes in a controlled fallback path, and never as an unconditional dependency for every user request.

The Problem

An e-commerce search system must return relevant products, filters, and discovery suggestions from a large product catalog. In this scenario, assume the catalog contains about 10 million products. Each product has fields such as name, description, brand, category, price, stock state, rating, and attributes.

The system has four practical requirements:

A naive LLM-first flow looks like this:

User search
  -> API Gateway
  -> Search Service
  -> LLM rewrite for every query
  -> Search index
  -> Product results

This design has three problems.

First, it adds variable latency to every request. Even if most queries are simple, they still wait for the model.

Second, it creates unbounded cost. A high-traffic query such as "running shoes" should not need repeated model reasoning.

Third, it creates a hard availability dependency. If the model provider is slow, rate limited, or unavailable, the search page degrades even for queries that a normal index could answer.

A better design treats LLM query understanding as an optimization layer, not the primary request path.

Target Architecture

The architecture has three cooperating flows:

Offline intelligence path
  Search logs
  -> Batch aggregation
  -> Query enrichment
  -> LLM segmentation and classification
  -> Semantic linking and query expansion
  -> Cached search plans

Online search path
  User query
  -> API Gateway
  -> Search Service
  -> Query cache lookup
  -> Keyword search plus vector search
  -> Lightweight re-ranking
  -> Product results

Controlled fallback path
  Low confidence or zero results
  -> LLM query rewrite
  -> Run search again
  -> Return rescued results

The key idea is to precompute expensive intelligence for common and emerging queries, then let the live service do fast retrieval. The LLM still helps with ambiguous searches, but it no longer blocks the majority of requests.

Step 1: Build the Offline Query Enrichment Pipeline

The offline pipeline converts search behavior into reusable search intelligence. It can run daily for comprehensive analysis and more frequently during high-traffic periods such as sales or festivals.

A practical pipeline looks like this:

Analytic data lake
  -> Spark aggregation job
  -> Query event queue
  -> Enrichment workers
  -> Query processing queue
  -> Segmentation and classification service
  -> Query expansion service
  -> Ranking and indexing workers
  -> Smart query cache

Aggregate Search Logs

The first job reads search events from the analytics store. Each event should connect a user query to what happened next: impressions, clicks, add-to-cart actions, and purchases.

The aggregation step should also count query frequency. Frequency decides which queries deserve the fastest cache tier later.

A simplified event model can be expressed without tying it to a specific storage format:

SearchEvent
  user_id
  session_id
  raw_query
  normalized_query
  displayed_products
  clicked_product
  added_to_cart_product
  purchased_product
  device_type
  event_time

The output is not just a list of queries. It is evidence about which products users actually found useful for those queries.

Enrich Each Query

Next, enrichment workers normalize and hydrate each query before model processing.

Useful enrichment fields include:

• Corrected spelling and normalized casing
• User purchase history and brand preferences
• Location, time of day, device type, and local trends
• Query frequency and conversion signals

This step should be parallelized behind a queue so log aggregation and enrichment can scale independently. If an enrichment worker fails, the rest of the pipeline can continue processing other batches.

Segment and Classify the Query

The segmentation service turns a natural language query into the platform's product taxonomy.

For example, a query such as "organic long-grain rice" might become:

The important guardrail is that the model should classify into known platform categories and attributes. It should not invent taxonomy paths that your catalog does not support.

Expand the Query Using Existing Products

An LLM cannot see a huge product catalog in one prompt because every model has a context window limit. The system must retrieve a small candidate set first.

That is where vector search helps. Product descriptions and tags are converted into embeddings, which are numerical representations of meaning. The query is also embedded, and the vector database returns products that are conceptually close even when they do not share keywords.

The expansion service should use this small candidate set to build a better structured search plan. It can include synonyms, alternatives, and related product types that actually exist in the catalog.

Step 2: Cache Search Plans, Not Full Product Lists

Caching the final list of product IDs looks attractive because it is the fastest possible read. It is also risky.

If a cached product goes out of stock, changes price, or receives a rating update, the user can see stale results. For search, freshness is part of correctness.

The safer default is to cache the structured search plan that the offline pipeline produced. At request time, the search service executes that plan against the live search index. This keeps price, stock, filters, and ratings current.

A hybrid strategy works best:

• Cache structured search plans for most precomputed queries.
• Cache final results only for a small set of very hot and stable queries.
• Use a short time-to-live for any final result cache.
• Refresh query plans when the offline pipeline produces better versions.

Step 3: Use Cache Tiers Based on Query Frequency

Not every query deserves the same cache. A few queries drive a large share of traffic, while the long tail contains many unique phrases.

Use three tiers:

The runtime lookup becomes:

Normalize query
  -> Check L1 exact cache
  -> Check L2 exact cache
  -> Embed query
  -> Check warm semantic cache
  -> Run fast hybrid search
  -> Use LLM fallback only if confidence is low

L1 cache should be warmed from a central hot-query list generated by the offline pipeline. At startup, each search service instance loads the current top queries into local memory. When the pipeline publishes a new hot list, running instances refresh their local cache.

This avoids a common operational mistake: letting each service instance guess what belongs in memory.

Step 4: Keep the Online Search Path Fast

The online service should assume most searches can be answered without a model call.

A fast path has three parts:

1. Keyword search against Elasticsearch or OpenSearch for exact terms, filters, and facets.
2. Semantic search against the vector database for intent matches.
3. A lightweight re-ranker that merges, deduplicates, and orders the candidate results.

The re-ranker can use signals such as keyword score, semantic score, product popularity, and user affinity. Personalization should be lightweight on the hot path. Expensive global analysis belongs in the offline job.

Here is a conceptual orchestrator. It avoids provider-specific details and focuses on the decision flow.

from dataclasses import dataclass
from typing import Sequence


@dataclass(frozen=True)
class SearchRequest:
    query: str
    user_id: str
    page: int
    filters: tuple[str, ...]


@dataclass(frozen=True)
class ProductHit:
    product_id: str
    title: str
    category: str
    score: float


def search_products(request: SearchRequest) -> Sequence[ProductHit]:
    normalized_query = normalize_query(request.query)

    cached_plan = find_cached_search_plan(normalized_query)
    if cached_plan is not None:
        candidates = execute_search_plan(cached_plan, request.filters)
        return rerank_for_user(candidates, request.user_id)

    fast_candidates = run_hybrid_fast_path(normalized_query, request.filters)
    fast_score = best_score(fast_candidates)

    if fast_score >= 0.65:
        return rerank_for_user(fast_candidates, request.user_id)

    rewritten_query = rewrite_query_with_llm(normalized_query)
    rescued_candidates = run_hybrid_fast_path(rewritten_query, request.filters)
    return rerank_for_user(rescued_candidates, request.user_id)

This design makes the LLM a P99 rescue path. It is called only when cache lookup and hybrid retrieval cannot produce a confident result.

That tradeoff is intentional. For normal traffic, users get sub-second behavior. For difficult long-tail searches, the system can spend a few seconds rewriting the query because a slower useful result is better than a fast empty page.

Step 5: Add Product Discovery Without Blocking Search

Search should not only find the requested product. It should also surface complementary items. A search for a cake mix might show frosting, candles, or baking trays. A search for hiking shoes might suggest socks or a rain jacket.

Do not compute those suggestions with an online LLM call.

Instead, let the offline pipeline combine segmented products with co-purchase analytics. The LLM can help choose complementary categories and product groups, then write the result into a discovery cache.

The online flow stays simple:

User searches
  -> Main search executes
  -> Discovery cache lookup runs in parallel
  -> Results page shows products plus related suggestions

This gives the user a richer product discovery experience without adding model latency to the main result path.

Step 6: Handle New Trends With a Near-Real-Time Speed Layer

A daily batch job is not enough during sudden demand shifts. A celebrity outfit, local weather change, sports event, or flash sale can create a new query pattern in minutes.

Use a near-real-time speed layer beside the daily batch layer:

Live query log topic
  -> Stream processor
  -> Sliding window count
  -> Trend detector
  -> Lite enrichment with fast model
  -> Staging warm cache
  -> Human approval for high-velocity trends
  -> Public cache

Daily batch layer
  -> Deep analysis
  -> Global ranking
  -> Overwrite temporary trend entries

The stream processor can use a 5-minute sliding window that updates every 10 seconds. If a normalized query crosses a trend threshold, such as more than 100 searches in 5 minutes, send only that query through a faster enrichment path.

The speed layer should skip slow, expensive steps. It should perform simple segmentation and write a good-enough entry to a staging cache. Later, the full batch job overwrites it with the higher-quality result.

For high-velocity trends, do not promote model-generated search plans directly to production. Send an approval alert to a merchandising or search-quality reviewer first. Viral queries amplify mistakes quickly.

Step 7: Choose the Vector Index for Your Catalog Size

Semantic search depends on nearest-neighbor lookup in a high-dimensional vector space. Exact nearest-neighbor search is often too slow at catalog scale, so production systems use approximate nearest neighbor techniques.

Two common choices are graph-based search and inverted partitioning.

For a 10 million product catalog, HNSW is a reasonable default because it favors low latency and high recall at the scale described here. If the catalog grows toward billions of items and memory becomes the dominant constraint, inverted partitioning becomes more attractive.

Step 8: Test Relevance Before Shipping Ranking Changes

Search changes should not ship only because the code passes unit tests. A ranking change can be technically correct and still reduce revenue.

Use three quality gates.

Golden Query Regression

A golden set contains high-value queries that must continue to work. For example, a query for a known brand should return that brand near the top. A query for a major category should return products from that category.

from dataclasses import dataclass


@dataclass(frozen=True)
class GoldenCase:
    query: str
    expected_value: str
    expected_field: str


GOLDEN_CASES = (
    GoldenCase("wireless earbuds", "Electronics", "category"),
    GoldenCase("trail running shoes", "Footwear", "category"),
    GoldenCase("vitamin drink", "Beverages", "category"),
    GoldenCase("noise cancelling headphones", "Audio", "category"),
)


def test_top_results_match_golden_cases():
    for case in GOLDEN_CASES:
        results = get_search_results(case.query, limit=3)
        assert results, "No results for golden query: " + case.query

        top = results[0]
        actual = getattr(top, case.expected_field)
        assert case.expected_value in actual

LLM-as-a-Judge on Samples

For broader coverage, a strong judge model can compare search results against the user query and assign a relevance score. Do not run this on live traffic. Run it on a controlled sample, such as a few hundred daily queries, so cost stays bounded.

Use the judge score as a directional quality signal, not as the only release gate. Human review is still needed for ambiguous, sensitive, or business-critical queries.

A/B Testing

Any major ranking or query expansion change should roll out to a subset of users first. Compare the treatment group against the existing search stack using click-through rate, conversion rate, and revenue per session.

import hashlib


def experiment_group(user_id: str, experiment_name: str) -> str:
    seed = (user_id + ":" + experiment_name).encode("utf-8")
    bucket = int(hashlib.sha256(seed).hexdigest(), 16) % 100
    if bucket < 50:
        return "control"
    return "treatment"


def handle_search(request: SearchRequest):
    group = experiment_group(request.user_id, "hybrid_rerank_v2")
    if group == "treatment":
        return search_products(request)
    return legacy_search(request)

The bucket function must be deterministic so the same user stays in the same group throughout the test.

Common Mistakes

• Calling the LLM for every query, even when a cache or hybrid search can answer it.
• Caching product IDs as the default value and serving stale prices or out-of-stock items.
• Forgetting to count query frequency during log aggregation, then having no reliable way to decide cache tiers.
• Treating long-tail semantic cache misses as normal failures instead of routing them through a controlled fallback path.
• Promoting near-real-time trend outputs directly to the public cache without review.
• Optimizing P99 for speed only, when a slower rescued result may be better than an immediate empty result.
• Measuring only latency and error rate while ignoring click-through rate and conversion rate.

Production Checklist

Before shipping LLM-assisted search, verify the following:

• The online search path can serve common queries without calling an LLM.
• Query plans are cached separately from final product results.
• Final result caches use short time-to-live values.
• L1 cache is loaded from a generated hot-query list.
• L2 cache handles the broader set of frequent queries.
• The warm tier supports semantic lookup for long-tail queries.
• The fallback LLM path triggers only on low-confidence or zero-result searches.
• Keyword and vector results are merged, deduplicated, and re-ranked.
• Product discovery suggestions are precomputed offline.
• Near-real-time trend entries are staged before public promotion.
• Ranking changes pass golden query regression tests.
• A/B tests monitor relevance and business metrics, not only service health.
• The GenAI service has timeouts, retries, circuit breakers, and model fallback.

Conclusion

LLMs can make e-commerce search smarter, but only if they are not placed blindly in the hot path. The scalable design is to move expensive query understanding into offline and near-real-time pipelines, cache the resulting search plans, and reserve live model calls for difficult low-confidence searches.

The result is a search system that can understand ambiguous user intent, preserve catalog freshness, handle product discovery, and stay responsive under high traffic. The important architectural move is simple: let LLMs build search intelligence, but let the search service serve most users from fast retrieval, caches, and lightweight ranking.