Serving Personalized Lessons at 350K Peak RPS Without Calling an LLM on Every Request

A language learning app has a tempting product requirement: make every lesson feel personalized. After a learner answers a few questions, the app should choose the next exercise based on their recent mistakes, known vocabulary, current skill strength, and target difficulty.

The risky implementation is to call a large language model (LLM) every time the learner needs the next question. That looks flexible in a prototype, but it fails under production constraints. The request path becomes slow, every learner action creates variable API cost, and a provider outage can break the learning flow. Worse, generating educational content live can introduce incorrect grammar, bad distractors, unsafe text, or inconsistent difficulty.

A production design should separate two jobs that are often confused:

• Creating lesson content at scale
• Selecting the right approved lesson for a learner in real time

The content creation path can use LLMs because it is offline, asynchronous, and reviewed by humans. The live serving path should be fast, cache-backed, and able to return a lesson even when the LLM layer is slow or unavailable.

The Problem

Assume the platform has these requirements:

• Learners need personalized lessons based on progress and mistakes.
• Admins and language experts need a way to generate and review new questions.
• The learner-facing API has a P99 latency target below 500 ms from answer submission to feedback or the next question.
• The system must stay available if an LLM provider has high latency, rate limits, or downtime.
• LLM calls must be controlled because cost grows with usage.
• The platform may need to handle about 350K peak requests per second after accounting for lesson actions, answer submissions, feedback, conversation practice, and analytics traffic.

A naive live flow looks like this:

Learner answers question
  |
  v
Lesson API calls LLM
  |
  v
LLM invents or selects next exercise
  |
  v
Lesson API returns question

This design couples user experience to the slowest and least predictable dependency in the system. It also puts quality control in the wrong place. Educational content should not become live just because a model generated it.

A better design uses two separate paths:

Offline content path:
Admin portal
  -> Lesson generation service
  -> Generation queue
  -> LLM orchestrator workers
  -> Generated questions staging area
  -> Human review portal
  -> Approved question bank

Online serving path:
Learner app
  -> API gateway
  -> Lesson curator service
  -> Redis playlist buffer
  -> Question bank
  -> Personalized lesson response

The LLM still adds value, but it is moved away from the hot path. The learner sees a fast API. The content team gets scale. The platform keeps a quality gate.

Scope and Main Components

This design has four main actors:

The system uses these components:

The key decision is simple: LLM-generated content is allowed into the system only through an offline review pipeline, while live personalization mostly chooses from approved content.

Estimate the Hot Path Before Designing It

The live system must be designed around traffic, not around the happy path demo.

A platform with 50 million daily active learners, 3 lessons per learner per day, and 10 questions per lesson needs to serve a large number of question interactions:

50,000,000 learners
* 3 lessons per learner per day
* 10 questions per lesson
/ 86,400 seconds per day
= about 17,340 question events per second

Each question can involve at least:
- Fetching question content
- Submitting an answer
- Fetching feedback or results

17,340 * 3 = about 51,000 average API requests per second

With a 5x peak factor:
51,000 * 5 = about 255,000 peak API requests per second

After adding conversation practice and analytics traffic:
about 350,000 peak requests per second

At that scale, calling an LLM for every next lesson is not just slow. It is also financially and operationally unstable. The design goal is to make the normal lesson request a cache read plus a question-bank lookup.

Build the Offline Content Factory

The offline content path creates new questions, but does not immediately expose them to learners.

A good admin workflow supports two modes:

• Small synchronous generation for quick experiments, usually up to 5 questions
• Large asynchronous generation for bulk jobs, such as 100 or more questions

For bulk jobs, the API should return immediately with an accepted job identifier. Workers do the slow generation later.

def submit_question_generation(request):
    validate_admin_request(request)

    if request.question_count <= 5:
        questions = generate_small_batch(request)
        save_to_staging(questions, status="IN_REVIEW")
        return SyncGenerationResult(questions=questions)

    parent_job = create_generation_job(
        language=request.language,
        level=request.level,
        topic=request.topic,
        question_format=request.question_format,
        total_count=request.question_count,
    )

    for batch in split_into_batches(request, batch_size=10):
        generation_jobs_queue.publish(
            GenerationTask(parent_job_id=parent_job.id, batch=batch)
        )

    return AcceptedGenerationJob(job_id=parent_job.id)

The async flow looks like this:

1. Admin requests 100 new exercises.
2. Lesson generation service validates the request.
3. Service creates one parent job.
4. Service splits work into batches of 10 questions.
5. Queue stores each batch as a separate task.
6. Orchestrator workers consume tasks and call the selected LLM.
7. Results are sent to a result queue.
8. Database workers write generated questions to staging.
9. Reviewers approve, edit, or reject questions.
10. A scheduled ingestion job promotes accepted questions into the main question bank.

This pattern gives you backpressure. If LLM calls slow down, the queue grows, but the admin API does not block until a browser request times out. It also gives you job status tracking, so the admin UI can show states such as queued, processing, completed, or failed.

Keep Human Review Between Generation and Serving

A generated exercise should land in a staging area first. Reviewers need enough metadata to judge whether the question is safe and useful:

• Language
• Difficulty level
• Topic or concept tags
• Question type
• Correct answer
• Distractors or answer choices
• Current review status
• Quality score from automated checks

The review portal should support three actions:

• Accept the question
• Reject the question
• Edit the text, choices, answer, tags, or difficulty

Only accepted records move into the main question bank. This keeps the LLM useful without making it the final authority on grammar, curriculum quality, or learner safety.

The ingestion job can run nightly or daily. Before promotion, it can run additional validations such as required field checks, duplicate detection, and format checks.

Serve Lessons From a Warm Redis Buffer

The live lesson request should usually avoid the LLM completely.

Instead of waiting until a learner asks for a lesson, precompute a short playlist of question IDs for that learner. Store that playlist in Redis as a list. When the learner starts a lesson, the curator service atomically pops the next question ID and fetches the full question from the question bank.

def start_lesson(user_id, skill_id):
    key = playlist_key(user_id=user_id, skill_id=skill_id)

    question_id = redis_client.left_pop(key)

    if question_id is None:
        lesson = select_cold_path_lesson(user_id=user_id, skill_id=skill_id)
        refill_requests.publish(RefillRequest(user_id=user_id, skill_id=skill_id))
        return lesson

    question = question_bank.get_by_id(question_id)
    return LessonResponse(question=question)

The atomic pop matters. If the same learner opens two sessions or retries a request, the same cached question ID should not be served twice from the same playlist buffer.

A warm path request should look like this:

Learner requests lesson
  |
  v
Lesson curator service
  |
  v
Redis LPOP from learner playlist
  |
  v
Question ID found
  |
  v
Fetch full question from approved question bank
  |
  v
Return lesson

This is the fast path. It is predictable, cheap, and does not depend on model latency.

Refill the Playlist Asynchronously

The playlist buffer must be refilled before it runs out. Trigger refill checks from learner activity, not only from cache misses.

A typical trigger is lesson completion. When a learner finishes a lesson, the service that handles that event publishes a lightweight refill check message. A background worker checks the learner's Redis playlist length. If fewer than 5 to 10 lessons remain, it starts a refill job.

The refill worker gathers:

• Recent progress
• Recent mistakes
• Skill strength
• Known vocabulary
• History from the last 50 lessons
• A candidate pool of unseen questions from the approved question bank

Then the orchestrator can ask an LLM to sequence 20 to 30 question IDs from the candidate pool. The model is not creating unreviewed content here. It is choosing and ordering approved items.

def maybe_refill_playlist(user_id, skill_id):
    key = playlist_key(user_id=user_id, skill_id=skill_id)

    if redis_client.list_length(key) >= 10:
        return RefillSkipped(reason="buffer_has_enough_items")

    context = user_context_service.load_profile(
        user_id=user_id,
        skill_id=skill_id,
        recent_lesson_limit=50,
    )

    candidates = question_bank.find_unseen_candidates(
        user_id=user_id,
        skill_id=skill_id,
        max_candidates=300,
    )

    personalization_queue.publish(
        PersonalizationJob(
            user_id=user_id,
            skill_id=skill_id,
            learner_context=context,
            candidate_questions=candidates,
            requested_playlist_size=30,
        )
    )

    return RefillQueued()

The orchestrator worker consumes the personalization job, chooses a model based on complexity and cost, validates that returned IDs belong to the candidate pool, and writes the new ordered IDs to Redis.

Lesson complete event
  |
  v
Refill check queue
  |
  v
Curation worker checks Redis buffer
  |
  v
If low, gather learner context and candidate questions
  |
  v
Personalization queue
  |
  v
LLM orchestrator sequences approved question IDs
  |
  v
Validated IDs are pushed into Redis playlist

This keeps the personalization engine decoupled from the learner's request. The user gets fast lesson starts, while expensive sequencing happens in the background.

Handle Cold Starts Without Blocking on the LLM

A cold path happens when Redis has no playlist for the learner. This is common for new users, returning users whose cache expired, or users after a Redis failure.

The cold path should not wait for a full LLM-powered refill. It should return one acceptable lesson using fast deterministic logic, then trigger async refill for future requests.

A practical fallback query should:

• Match the learner's language and skill
• Use the current difficulty level or skill strength
• Exclude questions seen in the last 20 lessons
• Prefer approved content with stable quality scores
• Return one lesson quickly

def select_cold_path_lesson(user_id, skill_id):
    profile = user_context_service.load_profile(user_id=user_id, skill_id=skill_id)

    recent_ids = lesson_history.load_recent_question_ids(
        user_id=user_id,
        limit=20,
    )

    return question_bank.find_first_match(
        language=profile.language,
        skill_id=skill_id,
        difficulty=profile.current_difficulty,
        excluded_question_ids=recent_ids,
    )

This lesson may be less personalized than a warm-path playlist, but it keeps the product usable. The user should not experience a blank screen because a background AI workflow has not finished.

Choose Storage Based on Access Patterns

Each data store should match the way the system reads and writes data.

The lesson history table should not keep unlimited hot data. Keep the recent window in the high-throughput store, such as the last 90 days, and stream older history into a data lake. The hot store supports fast duplication checks and recent personalization. The cold store supports analytics and model improvement.

Redis should not be the source of truth. It is a rebuildable performance layer. If Redis loses cached playlists, the system can fall back to cold-path lesson selection and refill playlists again.

Route LLM Calls Through an Orchestrator

The platform should not let every service call every model directly. Use an orchestrator as a centralized LLM gateway.

The orchestrator has four jobs:

1. Build prompts with the right context, constraints, and expected response shape.
2. Choose the cheapest model that is good enough for the task.
3. Apply retries, circuit breakers, and provider fallbacks.
4. Validate the response before another service uses it.

A model router might follow rules like these:

This prevents a simple operation from using the most expensive model by default.

The resilience layer should handle common provider failures:

• Retry transient 429 or 503 responses with exponential backoff.
• Cap retries to avoid adding too much latency.
• Open the circuit when a provider repeatedly fails or times out.
• Send traffic to a backup model while the primary provider recovers.
• Test the primary provider later with a small probe before closing the circuit.

For learner-facing flows, prefer fallback over long retries. A 30-second retry loop is effectively downtime for someone waiting for the next question.

Use Deterministic Tools for Grading

LLMs are useful for friendly explanations, but they should not be trusted for precise grading when deterministic code can do the job. For example, if a learner submits a numerical answer, do not ask the model to decide whether it is correct. Ask the model to call a grading tool, or have the application call the tool directly.

def grade_numeric_answer(user_answer, expected_formula, variables):
    expected_value = safe_formula_runner.evaluate(
        formula=expected_formula,
        variables=variables,
    )

    is_correct = numbers_are_close(
        left=user_answer,
        right=expected_value,
        tolerance=0.001,
    )

    return GradeResult(
        correct=is_correct,
        expected_value=expected_value,
    )

The LLM can then turn the tool result into a helpful message, such as explaining the mistake or encouraging the learner. The correctness decision stays deterministic.

Scale the Services and Queues

The curation service and orchestrator workers should scale horizontally.

Use different scaling signals for different components:

For the question bank, cache lesson objects by question ID because warm-path playlist entries contain IDs, not full lesson content. Add read replicas because the approved question bank is read-heavy.

For user progress, use a store that can absorb high write throughput. Lesson completions, answer submissions, and feedback events are frequent and should not overload the content database.

For Redis, use clustering and lightweight persistence for recovery, but design as if cached playlists can disappear. The cold path is your safety net.

Test the Failure Modes

Testing should target the exact risks created by this architecture.

Load testing

Simulate users starting lessons, submitting answers, and requesting feedback. Increase traffic until latency or queue depth breaks your target. The goal is not only to find max throughput. It is to verify that autoscaling reacts before users feel the slowdown.

Queue testing

Create a burst of admin generation jobs. Confirm that:

• The admin API returns accepted responses quickly.
• Queue depth increases temporarily.
• Workers scale out.
• Parent jobs complete only after all child batches finish.
• Failed child batches do not mark the parent job as successful.

Redis cache testing

Force playlist cache misses and verify that the cold path returns a lesson. Then confirm that refill jobs are created and later requests use the warm path again.

LLM dependency testing

Simulate an LLM provider that becomes slow, returns rate limits, or returns server errors. The test passes only if the circuit breaker opens and the orchestrator reroutes or falls back without breaking learner-facing requests.

AI quality regression testing

Keep a dataset of common learning interactions and expected high-quality responses. Run it whenever prompts or models change. Check accuracy, safety, tone, and whether generated content still matches the requested language level and format.

Monitor the Signals That Predict User Pain

The most useful alerts are the ones that tell you a fast path is becoming a slow path.

Track these metrics:

A drop in playlist hit rate is especially important. It means users are not benefiting from precomputation. Even if the API still returns a lesson, personalization quality and latency may degrade.

Common Mistakes

• Calling the LLM during every lesson request.
• Letting generated content go live without expert review.
• Treating Redis playlists as durable learner state.
• Using one expensive model for every generation, sequencing, and feedback task.
• Retrying LLM calls too aggressively in user-facing flows.
• Forgetting to trigger async refill after a cold-path response.
• Updating skill strength synchronously on every answer instead of using background aggregation.
• Storing unlimited lesson history in the hot operational database.
• Monitoring average latency while ignoring P99 latency and cache hit rate.

Implementation Checklist

• Split content generation from live lesson serving.
• Put generated questions into staging first.
• Require human review before promotion to the question bank.
• Use async jobs for large generation requests.
• Split large generation jobs into smaller worker tasks.
• Store approved content in a read-optimized question bank.
• Store recent learner activity in a high-throughput progress store.
• Maintain Redis playlist buffers per learner and skill.
• Serve warm-path lessons with an atomic pop from Redis.
• Use a fast rule-based cold path on cache miss.
• Trigger background refill after lesson completion and cache miss.
• Route all model calls through an orchestrator.
• Add retries, circuit breakers, and model fallbacks.
• Use deterministic grading tools where exact correctness matters.
• Test cache misses, provider degradation, queue backlogs, and model quality regressions.
• Alert on P99 latency, Redis hit rate, queue depth, job duration, and provider errors.

Conclusion

The reliable way to personalize lessons with LLMs is not to make the LLM responsible for every live decision. Use it where latency is acceptable and quality can be reviewed: offline content generation and background sequencing of approved question IDs.

The learner-facing path should be boring in the best way: authenticate the request, pop a precomputed lesson ID from Redis, fetch approved content, and return quickly. When the cache is empty, serve a safe rule-based fallback and refill in the background.

That split gives you personalization without making every learner wait on model inference, and it gives the engineering team clear places to scale, test, monitor, and recover from failure.