I-Prompt Rate Limits & Batching: I-API yakho ye-LLM ine-speed limit (ngaphandle kwe-product yakho) Uyavumela i-function, i-traffic yakho i-spikes, kwaye ngexesha elifutshane i-LLM yakho uqala ukuguqulwa Njengoko ukunikezela iiphakheji zokuhamba. 429s Iindaba ebuthayo: Izimali zentengiso zihlanganisa. Iindaba ezinhle: —kwenziwe ngokuvamile nge-prompts ezininzi, i-traffic ebandayo, kunye ne-output formats eyenziwe ngempumelelo kwi-scale. most LLM “rate limit incidents” are self-inflicted Le nqaku kuyinto playbook esebenzayo: 
 
 
 
 ukufumana iingxaki ezinxulumene neengxaki, ukunceda izindlela ezivamile ze-failure, kunye Ukudibanisa imibuzo ngaphandle kokuguqula imibuzo yakho kwi-soup. 1) I-Three Limits You Really Hit (NgoLula Yintoni) Iinkonzo ezahlukeneyo ziquka izinto ezahlukeneyo, kodwa izixhobo zihlanganisa: 1.1 I-Context window (i-max tokens ngalinye isicelo) Ukuba unayo ifumaneka kwi-model context window, isivumelwano isivumelwano ngokushesha. input + output Ukucaciswa: 
 
 
 “Ukuhlukaniswa ubude obungapheliyo” “Izixhobo zayo zithunyelwe kwi-X tokens...” Ukucaciswa: 
 
 Ukuqhathanisa, ukuqhathanisa, okanye ukucacisa idatha. 1.2 RPM (Imininingwane ngenyanga) Ingaba ungenza kwi-token limits kwaye ungenza ukuba utshintshe iintlobo ezininzi. I-Gemini ibhokisi ngokubanzi i-RPM njenge-core dimension. Ukucaciswa: 
 
 
 “I-rate limit eyenziwa kwi-requests per minute” I-HTTP ye-429 Ukucaciswa: 
 
 I-client-side pacing, iingxowa kunye ne-backoff. 1.3 I-TPM / i-Token Passport Limits Iimpawu ze-Anthropic rate limits (ITPM/OTPM). Gemini ngokufanelekileyo ibonisa i-token-per-minute njenge-dimension key. RPM + input tokens/minute + output tokens/minute Ukucaciswa: 
 
 
 “I-rate limit eyenziwe ngexesha lokusetyenziswa kwe-token ngenyanga” I-429 + i-Retry-After header (i-Anthropic ibiza oku) Ukucaciswa: 
 
 ukunciphisa i-tokens, i-batch ngempumelelo, okanye ukhangela i-quota ephezulu. 2) Iimveliso ezininzi eziqhelekileyo "I-Prompt Limit" Ukukhangisa 2.1 I-"one prompt to rule them all" anti-pattern Ufuna ukuba: 
 
 
 
 
 
 
 Ukucaciswa Ukuhlaziywa Ukubhalisa Ukucaciswa Ukucaciswa I-Logic ye-Business ...kunye isicelo elinye, yaye ngoko ufuna ukuba iintlobo zokusetyenziswa kwe-token zihlala. Ukuba ufuna i-Logic Multi-Step, usebenzise (I-Pompts ezincinane kunye ne-intermediate outputs eyenziwe). Split the workflow Prompt Chaining I-Bursty Traffic (i-silent RPM killer) I-Cron jobs, i-retries, i-user clicks, i-webhook bursts - yonke into ivela kwi-minute enhle kakhulu. Ukuba i-client yakho inikeza imibuzo njenge-machine gun, i-provider yakho uyibophelela njenge-bouncer. 2.3 I-output ye-unstructured = i-parsing emangalisayo Ukuba i-output yakho i-"kinda JSON-ish", i-parser yakho ivula i-therapist epheleleyo. Yenza Model Output okanye i-table fixed. Qhagamshelana ne-format njenge-contract. strict JSON 3) I-Rate Limit Survival Kit (i-Compliant, i-Practical, i-Nudge) 3.1 Prompt-Side: ukunciphisa i-tokens ngaphandle kokuzalwa i-signal 
 
 
 
 Ukugqiba i-marketing fluff (iimodeli akufuneka umzekelo wokufundisa inkampani yakho). Ukuguqula i-boilerplate ebandayo kwi-"policy block" esincinane kwaye isetyenziswe. Prefer fields over prose (“i-material=316 stainless steel” ibandakanya i-paragraph). I-Prompt ye-Rewrite ye-Prompt ye-Prompt ye-Prompt ye-Prompt ye-Prompt ye-Prompt ye-Prompt ye-30-50% Before (chatty): 
 
 "Ndiya i-smart home brand eyasungulwa ngo-2010 ... nceda uqhagamshelane iintlobo ze-3 ze-marketing..." "Ndiya i-smart home brand eyasungulwa ngo-2010 ... nceda uqhagamshelane iintlobo ze-3 ze-marketing..." After (dense + precise): 
 
 “Write 3 UK e-commerce lines. Product: smart bulb. Material=PC flame-retardant. Feature=3 iintlobo lokushisa. Audience=living room.” “Write 3 UK e-commerce lines. Product: smart bulb. Material=PC flame-retardant. Feature=3 iintlobo lokushisa. Audience=living room.” 3.2 I-Request-Side: i-backoff njenge-adult Ukuba umncedisi waya I-Anthropic ivumela ngokucacileyo i-Retry-After kwi-429s. Retry-After Ukusetyenziswa kwe-Backoff + Jitter: 
 
 
 
 
 Ukucinga 1: 1s Ukucinga 2: 2–3s Ukucinga 3: 4–6s Ngokuqhelekanga ngokushesha 3.3 Ukusetyenziswa kwe-System-Side: i-Cheve + i-Concurrency Caps Ukuba i-akhawunti yakho inokusetyenziswa kwi-10 iingcebiso ezisetyenziswa ngexesha elifanelekileyo, ayidlulisa i-200 iingcebiso kunye ne-"hope". Ukusebenzisa: 
 
 
 
 Izixhobo zokusebenza I-semaphore ye-competition kunye ne-rate limiter ye-RPM / TPM 4) Batching: Umgangatho okusheshayo yokunciphisa iingxowa, ixabiso, kunye ne-429s I-Batching ibonakala ngokuba: . one API request handles multiple independent tasks Ukusebenza kakuhle xa imisebenzi ziyafumaneka: 
 
 
 
 uhlobo efanayo (isib. i-20 iimveliso blurbs) i-independent (i-no step depends on another) I-Same Output Scheme Yintoni kunceda 
 
 
 
 Ngaphansi kwe-network round-trips izicelo ezincinane → ingcindezi RPM engaphansi Ukubuyekezwa okungapheliyo Kwakhona: Iiphakamiso ze-OpenAI zihlanganisa ngokubanzi i- "Batch API price" ye-column kwiimodeli ezininzi. (Ezi akufanele "batching free", kodwa kunceda kakhulu ukuba i-ecosystem ihamba le pattern.) 5) I-Batching Prompt Template Enye Ayikho Kwakhona Nazi ifomati ebonakalayo phantsi komgangatho. 5.1 Ukusetyenziswa kwiblokhi yeengxaki + isisombululo eshushu ye-JSON SYSTEM: You output valid JSON only. No Markdown. No commentary.
​
USER:
You will process multiple tasks. 
Return a JSON array. Each item must be:
{
  "task_id": <int>,
  "title": <string>,
  "bullets": [<string>, <string>, <string>]
}
​
Rules:
- UK English spelling
- Title ≤ 12 words
- 3 bullets, each ≤ 18 words
- If input is missing: set title="INSUFFICIENT_DATA" and bullets=[]
​
TASKS:
### TASK 1
product_name: Insulated smart mug
material: 316 stainless steel
features: temperature alert, 7-day battery
audience: commuters
​
### TASK 2
product_name: Wireless earbuds
material: ABS shock-resistant
features: ANC, 24-hour battery
audience: students
 Kwaye "INSUFFICIENT_DATA" i-clausule yakho yokuzonwabisa. Ixesha elinye elidlulileyo akufuneka ukuchithisa zonke iintlobo. 6) Ukusebenza kwePython: Batch → Call → Parse (Nge Guardrails) Okulandelayo i-moder-ish pattern enokufanelekileyo (i-provider SDK zihlanganisa, ngoko ukuthetha njengoko , ayikho inkcazelo ye-copy-paste). structure import json
import random
import time
from typing import Any, Dict, List, Tuple
​
MAX_RETRIES = 4
​
def backoff_sleep(attempt: int, retry_after: float | None = None) -> None:
    if retry_after is not None:
        time.sleep(retry_after)
        return
    base = 2 ** attempt
    jitter = random.random()
    time.sleep(min(10, base + jitter))
​
def build_batch_prompt(tasks: List[Dict[str, str]]) -> str:
    header = (
        "You output valid JSON only. No Markdown. No commentary.\n\n"
        "Return a JSON array. Each item must be:\n"
        "{\n  \"task_id\": <int>,\n  \"title\": <string>,\n  \"bullets\": [<string>, <string>, <string>]\n}\n\n"
        "Rules:\n"
        "- UK English spelling\n"
        "- Title ≤ 12 words\n"
        "- 3 bullets, each ≤ 18 words\n"
        "- If input is missing: set title=\"INSUFFICIENT_DATA\" and bullets=[]\n\n"
        "TASKS:\n"
    )
​
    blocks = []
    for t in tasks:
        blocks.append(
            f"### TASK {t['task_id']}\n"
            f"product_name: {t.get('product_name','')}\n"
            f"material: {t.get('material','')}\n"
            f"features: {t.get('features','')}\n"
            f"audience: {t.get('audience','')}\n"
        )
    return header + "\n".join(blocks)
​
def parse_json_strict(text: str) -> List[Dict[str, Any]]:
    # Hard fail if it's not JSON. This is intentional.
    return json.loads(text)
​
def call_llm(prompt: str) -> Tuple[str, float | None]:
    """Return (text, retry_after_seconds). Replace with your provider call."""
    raise NotImplementedError
​
def run_batch(tasks: List[Dict[str, str]]) -> List[Dict[str, Any]]:
    prompt = build_batch_prompt(tasks)
​
    for attempt in range(MAX_RETRIES):
        try:
            raw_text, retry_after = call_llm(prompt)
            return parse_json_strict(raw_text)
        except json.JSONDecodeError:
            # Ask the model to repair formatting in a second pass (or log + retry)
            prompt = (
                "Fix the output into valid JSON only. Preserve meaning.\n\n"
                f"BAD_OUTPUT:\n{raw_text}"
            )
            backoff_sleep(attempt)
        except Exception as e:
            # If your SDK exposes HTTP status + retry-after, use it here
            backoff_sleep(attempt)
            last_error = e
​
    raise RuntimeError(f"Batch failed after retries: {last_error}")
 Yintoni ukuguqulwa kwi-snippets 'classic'? 
 
 
 
 Thola i-JSON njengoko i-hard contract. Thumela ukuguqulwa kweformats ngokucacileyo (kuvela ngexabiso). I-centralized i-backoff logic ukuze zonke iingxowa zihlala ngexesha elifanayo. 7) Indlela Khetha Usayizi Batch (Ukufundisa Wonke Umgaqo) Ubungakanani batch iyahambelana nge: 
 
 
 
 
 ifenisha le-context (max tokens ngalinye isicelo) TPM Ukupholisa Ukusabela ukhuseleko Parsing inkcazelo yakho yebhizinisi kwi-"one batch failed" Ukusebenza kwe-heuristic: 
 
 
 
 Ukuqala nge-10-20 iimveliso kwa batch Ukusetyenziswa kweToken increase until you see:
 
 
 
 
 
 output format drift, or timeouts / latency spikes, or context overflow risk Nceda ushiye . max batch token budget 8) "i-Cost Math" ngaphandle kweenombolo ze-Fantasy Iimveliso ezintsha. Iimveliso ezintsha. Iimveliso ezintsha. Ngoko ke, ngexesha yokubhalisa iindidi ezidlulileyo ze-1K token, ukucacisa ixabiso usebenzisa inkcazelo le-price page ye-providers. I-OpenAI inikeza ixabiso ye-token kwi-API pricing page yayo. I-Anthropic inikeza i-pricing kunye ne-document rate limit levels. I-Cost Estimator esebenzayo: cost ≈ (input_tokens * input_price + output_tokens * output_price) / 1,000,000
 Emva koko, ukwandisa iimvavanyo eziqhagamshelwano: 
 
 
 
 Shrink Input I-Token Ukunciphisa i-output tokens Ukunciphisa inani lomsebenzisi (batch) 9) Iingxaki ze-batching (Ngoya ukuba awukwazi ukuza) Umthamo 1: Umthamo omncinci omnye umthamo omncinci “INSUFFICIENT_DATA” fallback ngalinye umsebenzi. Fix: Umthamo 2: i-output format i-drift ibandakanya i-parsing i-JSON, i-repair step, kunye ne-logging. Fix: I-Risk 3: I-batch iphezulu kakhulu → i-context overflow I-token budgeting + i-auto-splitting. Fix: Umthamo we-4: Iimpumelelo ze-"creative" zokuhamba amaxabiso Ukuba ufuna i-capacity engaphezulu, nqakraza i-limits engaphezulu kwaye nqakraza iimeko ze-provider. Fix: Ukucinga I-rate limits ayikho i-enemy. Yintoni i-system yakho yokuhambisa ngokukhawuleza: 
 
 
 
 iimveliso ezininzi ezininzi, Ukuhambelana okuninzi, okanye i-architecture yakho ibonelela i-”infinite throughput”. Ukuba usebenzise iintlawulo ezifana neengxaki ze-payloads (ayikwazanga i-prose), ukongeza i-pacing, kunye ne-batch njenge-adult-up, uya kufumana: 
 
 
 
 Ngaphansi 429s Iindleko ezincinci kunye nenkqubo ebandayo ngaphandle kwe-drama Yinto yonke umdlalo.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Read My Stories

Le audio iveliswe ngolwimi lokuqala lwebali!

I-Prompt Rate Limits & Batching: Indlela yokuvimbela i-API yakho ye-LLM ukususela

About Author

IZIMVO

ZIJONGE IIMPAWU

ELI NQAKU LINIKEZELWE KU

Related Stories

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps