Prompt Rate Limits & Batching: Your LLM API Has a Speed Limit (Even If Your Product Doesn’t) You ship a feature, your traffic spikes, and suddenly your LLM layer starts returning 429s like it’s handing out parking tickets. 429s The bad news: rate limits are inevitable. The good news: most LLM “rate limit incidents” are self-inflicted—usually by oversized prompts, bursty traffic, and output formats that are impossible to parse at scale. most LLM “rate limit incidents” are self-inflicted This article is a practical playbook for: understanding prompt-related throttles,
avoiding the common failure modes, and
batching requests without turning your responses into soup. understanding prompt-related throttles, avoiding the common failure modes, and batching requests without turning your responses into soup. 1) The Three Limits You Actually Hit (And What They Mean) Different providers name things differently, but the mechanics are consistent: 1.1 Context window (max tokens per request) If your input + output exceeds the model context window, the request fails immediately. input + output Symptoms: “Maximum context length exceeded”
“Your messages resulted in X tokens…” “Maximum context length exceeded” “Your messages resulted in X tokens…” Fix: shorten, summarise, or chunk data. shorten, summarise, or chunk data. 1.2 RPM (Requests Per Minute) You can be under token limits and still get throttled if you burst too many calls. Gemini explicitly documents RPM as a core dimension. Symptoms: “Rate limit reached for requests per minute”
HTTP 429 “Rate limit reached for requests per minute” HTTP 429 Fix: client-side pacing, queues, and backoff. client-side pacing, queues, and backoff. 1.3 TPM / Token throughput limits Anthropic measures rate limits in RPM + input tokens/minute + output tokens/minute (ITPM/OTPM).    Gemini similarly describes token-per-minute as a key dimension. RPM + input tokens/minute + output tokens/minute Symptoms: “Rate limit reached for token usage per minute”
429 + Retry-After header (Anthropic calls this out) “Rate limit reached for token usage per minute” 429 + Retry-After header (Anthropic calls this out) Fix: reduce tokens, batch efficiently, or request higher quota. reduce tokens, batch efficiently, or request higher quota. 2) The Most Common “Prompt Limit” Failure Patterns 2.1 The “one prompt to rule them all” anti-pattern You ask for: extraction
classification
rewriting
validation
formatting
business logic extraction classification rewriting validation formatting business logic …in a single request, and then you wonder why token usage spikes. Split the workflow. If you need multi-step logic, use Prompt Chaining (small prompts with structured intermediate outputs). Split the workflow Prompt Chaining 2.2 Bursty traffic (the silent RPM killer) Production traffic is spiky. Cron jobs, retries, user clicks, webhook bursts—everything aligns in the worst possible minute. If your client sends requests like a machine gun, your provider will respond like a bouncer. 2.3 Unstructured output = expensive parsing If your output is “kinda JSON-ish”, your parser becomes a full-time therapist. Make the model output strict JSON or a fixed table. Treat format as a contract. strict JSON 3) Rate Limit Survival Kit (Compliant, Practical, Boring) 3.1 Prompt-side: shrink tokens without losing signal Delete marketing fluff (models don’t need your company origin story).
Convert repeated boilerplate into a short “policy block” and reuse it.
Prefer fields over prose (“material=316 stainless steel” beats a paragraph). Delete marketing fluff (models don’t need your company origin story). Delete marketing fluff Convert repeated boilerplate into a short “policy block” and reuse it. Prefer fields over prose (“material=316 stainless steel” beats a paragraph). fields A tiny prompt rewrite that usually saves 30–50% Before (chatty): Before (chatty): “We’re a smart home brand founded in 2010… please write 3 marketing lines…” “We’re a smart home brand founded in 2010… please write 3 marketing lines…” After (dense + precise): After (dense + precise): “Write 3 UK e-commerce lines. Product: smart bulb. Material=PC flame-retardant. Feature=3 colour temperatures. Audience=living room.” “Write 3 UK e-commerce lines. Product: smart bulb. Material=PC flame-retardant. Feature=3 colour temperatures. Audience=living room.” 3.2 Request-side: backoff like an adult If the provider returns Retry-After, respect it. Anthropic explicitly returns Retry-After on 429s. Retry-After Use exponential backoff + jitter: attempt 1: 1s
attempt 2: 2–3s
attempt 3: 4–6s
then fail gracefully attempt 1: 1s attempt 2: 2–3s attempt 3: 4–6s then fail gracefully 3.3 System-side: queue + concurrency caps If your account supports 10 concurrent requests, do not run 200 coroutines and “hope”. Use: a work queue
a semaphore for concurrency
and a rate limiter for RPM/TPM a work queue work queue a semaphore for concurrency semaphore and a rate limiter for RPM/TPM rate limiter 4) Batching: The Fastest Way to Cut Calls, Cost, and 429s Batching means: one API request handles multiple independent tasks. one API request handles multiple independent tasks It works best when tasks are: same type (e.g., 20 product blurbs)
independent (no step depends on another)
same output schema same type (e.g., 20 product blurbs) independent (no step depends on another) same output schema Why it helps fewer network round-trips
fewer requests → lower RPM pressure
more predictable throughput fewer network round-trips fewer requests → lower RPM pressure more predictable throughput Also: OpenAI’s pricing pages explicitly include a “Batch API price” column for several models.  (That doesn’t mean “batching is free”, but it’s a strong hint the ecosystem expects this pattern.) 5) The Batching Prompt Template That Doesn’t Fall Apart Here’s a format that stays parseable under pressure. 5.1 Use task blocks + a strict JSON response schema SYSTEM: You output valid JSON only. No Markdown. No commentary.
​
USER:
You will process multiple tasks. 
Return a JSON array. Each item must be:
{
  "task_id": <int>,
  "title": <string>,
  "bullets": [<string>, <string>, <string>]
}
​
Rules:
- UK English spelling
- Title ≤ 12 words
- 3 bullets, each ≤ 18 words
- If input is missing: set title="INSUFFICIENT_DATA" and bullets=[]
​
TASKS:
### TASK 1
product_name: Insulated smart mug
material: 316 stainless steel
features: temperature alert, 7-day battery
audience: commuters
​
### TASK 2
product_name: Wireless earbuds
material: ABS shock-resistant
features: ANC, 24-hour battery
audience: students SYSTEM: You output valid JSON only. No Markdown. No commentary.
​
USER:
You will process multiple tasks. 
Return a JSON array. Each item must be:
{
  "task_id": <int>,
  "title": <string>,
  "bullets": [<string>, <string>, <string>]
}
​
Rules:
- UK English spelling
- Title ≤ 12 words
- 3 bullets, each ≤ 18 words
- If input is missing: set title="INSUFFICIENT_DATA" and bullets=[]
​
TASKS:
### TASK 1
product_name: Insulated smart mug
material: 316 stainless steel
features: temperature alert, 7-day battery
audience: commuters
​
### TASK 2
product_name: Wireless earbuds
material: ABS shock-resistant
features: ANC, 24-hour battery
audience: students That “INSUFFICIENT_DATA” clause is your lifesaver. One broken task shouldn’t poison the whole batch. 6) Python Implementation: Batch → Call → Parse (With Guardrails) Below is a modern-ish pattern you can adapt (provider SDKs vary, so treat it as structure, not a copy‑paste guarantee). structure import json
import random
import time
from typing import Any, Dict, List, Tuple
​
MAX_RETRIES = 4
​
def backoff_sleep(attempt: int, retry_after: float | None = None) -> None:
    if retry_after is not None:
        time.sleep(retry_after)
        return
    base = 2 ** attempt
    jitter = random.random()
    time.sleep(min(10, base + jitter))
​
def build_batch_prompt(tasks: List[Dict[str, str]]) -> str:
    header = (
        "You output valid JSON only. No Markdown. No commentary.\n\n"
        "Return a JSON array. Each item must be:\n"
        "{\n  \"task_id\": <int>,\n  \"title\": <string>,\n  \"bullets\": [<string>, <string>, <string>]\n}\n\n"
        "Rules:\n"
        "- UK English spelling\n"
        "- Title ≤ 12 words\n"
        "- 3 bullets, each ≤ 18 words\n"
        "- If input is missing: set title=\"INSUFFICIENT_DATA\" and bullets=[]\n\n"
        "TASKS:\n"
    )
​
    blocks = []
    for t in tasks:
        blocks.append(
            f"### TASK {t['task_id']}\n"
            f"product_name: {t.get('product_name','')}\n"
            f"material: {t.get('material','')}\n"
            f"features: {t.get('features','')}\n"
            f"audience: {t.get('audience','')}\n"
        )
    return header + "\n".join(blocks)
​
def parse_json_strict(text: str) -> List[Dict[str, Any]]:
    # Hard fail if it's not JSON. This is intentional.
    return json.loads(text)
​
def call_llm(prompt: str) -> Tuple[str, float | None]:
    """Return (text, retry_after_seconds). Replace with your provider call."""
    raise NotImplementedError
​
def run_batch(tasks: List[Dict[str, str]]) -> List[Dict[str, Any]]:
    prompt = build_batch_prompt(tasks)
​
    for attempt in range(MAX_RETRIES):
        try:
            raw_text, retry_after = call_llm(prompt)
            return parse_json_strict(raw_text)
        except json.JSONDecodeError:
            # Ask the model to repair formatting in a second pass (or log + retry)
            prompt = (
                "Fix the output into valid JSON only. Preserve meaning.\n\n"
                f"BAD_OUTPUT:\n{raw_text}"
            )
            backoff_sleep(attempt)
        except Exception as e:
            # If your SDK exposes HTTP status + retry-after, use it here
            backoff_sleep(attempt)
            last_error = e
​
    raise RuntimeError(f"Batch failed after retries: {last_error}") import json
import random
import time
from typing import Any, Dict, List, Tuple
​
MAX_RETRIES = 4
​
def backoff_sleep(attempt: int, retry_after: float | None = None) -> None:
    if retry_after is not None:
        time.sleep(retry_after)
        return
    base = 2 ** attempt
    jitter = random.random()
    time.sleep(min(10, base + jitter))
​
def build_batch_prompt(tasks: List[Dict[str, str]]) -> str:
    header = (
        "You output valid JSON only. No Markdown. No commentary.\n\n"
        "Return a JSON array. Each item must be:\n"
        "{\n  \"task_id\": <int>,\n  \"title\": <string>,\n  \"bullets\": [<string>, <string>, <string>]\n}\n\n"
        "Rules:\n"
        "- UK English spelling\n"
        "- Title ≤ 12 words\n"
        "- 3 bullets, each ≤ 18 words\n"
        "- If input is missing: set title=\"INSUFFICIENT_DATA\" and bullets=[]\n\n"
        "TASKS:\n"
    )
​
    blocks = []
    for t in tasks:
        blocks.append(
            f"### TASK {t['task_id']}\n"
            f"product_name: {t.get('product_name','')}\n"
            f"material: {t.get('material','')}\n"
            f"features: {t.get('features','')}\n"
            f"audience: {t.get('audience','')}\n"
        )
    return header + "\n".join(blocks)
​
def parse_json_strict(text: str) -> List[Dict[str, Any]]:
    # Hard fail if it's not JSON. This is intentional.
    return json.loads(text)
​
def call_llm(prompt: str) -> Tuple[str, float | None]:
    """Return (text, retry_after_seconds). Replace with your provider call."""
    raise NotImplementedError
​
def run_batch(tasks: List[Dict[str, str]]) -> List[Dict[str, Any]]:
    prompt = build_batch_prompt(tasks)
​
    for attempt in range(MAX_RETRIES):
        try:
            raw_text, retry_after = call_llm(prompt)
            return parse_json_strict(raw_text)
        except json.JSONDecodeError:
            # Ask the model to repair formatting in a second pass (or log + retry)
            prompt = (
                "Fix the output into valid JSON only. Preserve meaning.\n\n"
                f"BAD_OUTPUT:\n{raw_text}"
            )
            backoff_sleep(attempt)
        except Exception as e:
            # If your SDK exposes HTTP status + retry-after, use it here
            backoff_sleep(attempt)
            last_error = e
​
    raise RuntimeError(f"Batch failed after retries: {last_error}") What changed vs “classic” snippets? We treat JSON as a hard contract.
We handle format repair explicitly (and keep it cheap).
We centralise backoff logic so every call behaves the same way. We treat JSON as a hard contract. hard contract We handle format repair explicitly (and keep it cheap). format repair We centralise backoff logic so every call behaves the same way. 7) How to Choose Batch Size (The Rule Everyone Learns the Hard Way) Batch size is constrained by: context window (max tokens per request)
TPM throughput
response parsing stability
your business tolerance for “one batch failed” context window (max tokens per request) TPM throughput response parsing stability your business tolerance for “one batch failed” A practical heuristic: start with 10–20 items per batch
measure token usage
increase until you see:

output format drift, or
timeouts / latency spikes, or
context overflow risk start with 10–20 items per batch 10–20 items per batch measure token usage increase until you see:

output format drift, or
timeouts / latency spikes, or
context overflow risk output format drift, or
timeouts / latency spikes, or
context overflow risk output format drift, or timeouts / latency spikes, or context overflow risk And always keep a max batch token budget. max batch token budget 8) “Cost Math” Without Fantasy Numbers Pricing changes. Tiers change. Models change. So instead of hard-coding ancient per-1K token values, calculate cost using the provider’s current pricing page. OpenAI publishes per‑token pricing on its API pricing pages.   Anthropic also publishes pricing and documents rate limit tiers. A useful cost estimator: cost ≈ (input_tokens * input_price + output_tokens * output_price) / 1,000,000 cost ≈ (input_tokens * input_price + output_tokens * output_price) / 1,000,000 Then optimise the variables you control: shrink input tokens
constrain output tokens
reduce number of calls (batch) shrink input tokens constrain output tokens reduce number of calls (batch) 9) Risks of Batching (And How to Not Get Burnt) Risk 1: one bad item ruins the batch Fix: “INSUFFICIENT_DATA” fallback per task. Fix: Risk 2: output format drift breaks parsing Fix: strict JSON, repair step, and logging. Fix: Risk 3: batch too big → context overflow Fix: token budgeting + auto-splitting. Fix: Risk 4: “creative” attempts to bypass quotas Fix: don’t. If you need more capacity, request higher limits and follow provider terms. Fix: Final Take Rate limits aren’t the enemy. They’re your early warning system that: prompts are too long,
traffic is too bursty,
or your architecture assumes “infinite throughput”. prompts are too long, traffic is too bursty, or your architecture assumes “infinite throughput”. If you treat prompts like payloads (not prose), add pacing, and batch like a grown-up, you’ll get: fewer 429s
lower cost
and a system that scales without drama fewer 429s lower cost and a system that scales without drama That’s the whole game.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Prompt Rate Limits & Batching: How to Stop Your LLM API From Melting Down

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Developer's Guide to Merging AI with the Spring Ecosystem

Request Throttling: A Cybersecurity Shield Against Overload and Abuse

100 Days of AI Day 3: Leveraging AI for Prompt Engineering and Inference

10 Tips to Take Your ChatGPT Prompts to the Next Level

100 Days of AI Day 2: Enhancing Prompt Engineering for ChatGPT

The Noonification: The Next Era of AI: Inside the Breakthrough GPT-4 Model (10/6/2023)

A Developer's Guide to Merging AI with the Spring Ecosystem

Request Throttling: A Cybersecurity Shield Against Overload and Abuse

100 Days of AI Day 3: Leveraging AI for Prompt Engineering and Inference

10 Tips to Take Your ChatGPT Prompts to the Next Level

100 Days of AI Day 2: Enhancing Prompt Engineering for ChatGPT

The Noonification: The Next Era of AI: Inside the Breakthrough GPT-4 Model (10/6/2023)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps