Errors, retries, rate limits and cost

When you run examples on your own computer, API calls almost never fail. Once you go live it's different: at peak times the server returns 429 to say you're making too many requests, some request hangs for a minute without a response, and now and then there's a 500. If your code isn't ready for this, a user sees a wall of error text, or a page that spins forever.

Then there's money. One buggy loop, one retry without a limit, and you can burn a month's budget overnight.

This lesson first reproduces the common errors, then writes a call function you can drop straight into a project.

Common errors

code/03-llm-apps/errors.py deliberately triggers three kinds of error:

print("== 1. 密钥错误")
try:
    OpenAI(api_key="sk-wrong", base_url=BASE_URL).chat.completions.create(model=MODEL, messages=HELLO)
except openai.AuthenticationError as e:
    print(f"{type(e).__name__}，状态码 {e.status_code}")

print("== 2. 模型名写错")
try:
    OpenAI(api_key=os.environ["LLM_API_KEY"], base_url=BASE_URL).chat.completions.create(model="deepseek-flsh", messages=HELLO)
except openai.APIStatusError as e:
    print(f"{type(e).__name__}，状态码 {e.status_code}，{e.message[:100]}")

print("== 3. 超时（故意把超时设成 0.5 秒，并关掉 SDK 自带的重试）")
start = time.time()
try:
    OpenAI(api_key=os.environ["LLM_API_KEY"], base_url=BASE_URL, timeout=0.5, max_retries=0).chat.completions.create(
        model=MODEL, messages=[{"role": "user", "content": "写一篇 800 字的文章"}], extra_body=NO_THINKING)
except openai.APITimeoutError as e:
    print(f"{type(e).__name__}，用了 {time.time() - start:.1f} 秒")

The output:

== 1. 密钥错误
AuthenticationError，状态码 401
== 2. 模型名写错
BadRequestError，状态码 400，Error code: 400 - {'error': {'message': 'The supported API model names are deepseek-flash, deepseek-
== 3. 超时（故意把超时设成 0.5 秒，并关掉 SDK 自带的重试）
APITimeoutError，用了 0.7 秒

The openai SDK turns different errors into different exception classes, so you can handle them by type. The common ones:

The pattern is simple: if the mistake is yours, retrying won't help; if it's the other side's or the network's, you can retry. The error for a wrong model name is friendly: it lists the supported model names, so you can just fix it.

Also, when your balance runs out, DeepSeek returns status code 402, which the SDK raises as a generic APIStatusError. This error shouldn't be retried either; it should tell you to top up.

What the SDK already does for you

Many people don't know the openai SDK retries automatically by default. I checked the source of the current version (3.14):

• The default is max_retries=2, so after a failure it tries at most 2 more times.
• It retries on timeouts, connection failures, and status codes 408, 409, 429 and everything 500 and above. When the server's response headers explicitly ask it to retry or not to, it follows that too.
• It waits between retries, starting at 0.5 seconds and doubling each time, up to 8 seconds.
• The default timeout is 600 seconds, of which at most 5 seconds is spent establishing the connection.

In other words, even if you write nothing, the odd 429 or 500 gets quietly retried away by the SDK. That's good, but there are two things to watch.

A 600-second timeout is far too long. Nobody waits 10 minutes on a web page. For ordinary chat, set the timeout to 30–60 seconds; for complex tasks with thinking on it can be longer. Just pass timeout=60 when you create the client.

You can't see retries happening. The SDK retries silently, so you don't know that one call actually took three tries and waited over ten seconds. When you're investigating "why is this so slow?", that information matters.

A call function you can put in a project

So I usually turn off the SDK's own retries and write my own call function, with retries, rate limiting and bookkeeping together:

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=BASE_URL,
    timeout=60,  # 单次请求最多等 60 秒
    max_retries=0,  # 关掉 SDK 自带的重试，由下面的函数统一处理，方便记录
)
RETRYABLE = (openai.RateLimitError, openai.APITimeoutError, openai.APIConnectionError, openai.InternalServerError)
limiter = threading.Semaphore(5)  # 同一时刻最多 5 个请求在路上
log_lock = threading.Lock()
LOG = Path("calls.jsonl")


def call_llm(messages, max_attempts=4, **kwargs):
    for attempt in range(1, max_attempts + 1):
        start = time.time()
        try:
            with limiter:
                response = client.chat.completions.create(model=MODEL, messages=messages, **kwargs)
        except RETRYABLE as e:
            if attempt == max_attempts:
                raise
            # 指数退避：1 秒、2 秒、4 秒……再加一点随机，避免大家同时重试
            wait = 2 ** (attempt - 1) + random.random()
            print(f"  第 {attempt} 次失败（{type(e).__name__}），{wait:.1f} 秒后重试")
            time.sleep(wait)
            continue
        record = {
            "time": time.strftime("%Y-%m-%d %H:%M:%S"),
            "model": response.model,
            "seconds": round(time.time() - start, 2),
            "prompt_tokens": response.usage.prompt_tokens,
            "completion_tokens": response.usage.completion_tokens,
            "cost_usd": round(cost_usd(response.usage, MODEL), 6),
            "attempts": attempt,
        }
        with log_lock:
            with LOG.open("a") as f:
                f.write(json.dumps(record, ensure_ascii=False) + "\n")
        return response

Let's go through it piece by piece.

Retry only the errors worth retrying. RETRYABLE lists the four exceptions that can be retried. Errors like 401 and 400 aren't in it, so they're raised straight away and you find out immediately.

Exponential backoff with random jitter. After the first failure it waits a little over 1 second, after the second a little over 2, after the third a little over 4. Doubling the wait gives the server time to recover: if the other side is overloaded, retrying once a second only makes things worse. The random part from random.random() prevents many requests failing at the same moment and then retrying at the same moment, hitting the server in wave after wave.

Cap the number of retries. If all 4 attempts fail, give up and raise the exception to the caller. Never write infinite retries.

Limit concurrency. threading.Semaphore(5) ensures at most 5 requests are in flight at once. Providers limit both concurrency and requests per minute (as of September 2026, DeepSeek's docs give a concurrency limit of 2,500 for deepseek-flash and 500 for deepseek-v4-pro), and holding yourself back first is better than waiting to be hit with 429. More importantly, it stops a bug in your code from firing off thousands of requests in an instant.

Log every call. Time, model, duration, input and output tokens, cost and number of attempts, written as one line of JSON appended to calls.jsonl. The cost uses cost_usd from lesson 4 of module 01. With this log you can answer questions like "what does this feature cost a day?", "which requests are especially slow?" and "how often do retries happen?". Lesson 3 of module 06 builds complete logging and monitoring on top of this.

Trying it: 20 concurrent requests

questions = [f"用一句话解释 HTTP 状态码 {code} 的含义。" for code in [200, 201, 204, 301, 302, 304, 400, 401, 403, 404,
                                                                  405, 408, 409, 418, 429, 500, 502, 503, 504, 505]]
with ThreadPoolExecutor(20) as pool:
    answers = list(pool.map(lambda q: call_llm([{"role": "user", "content": q}], extra_body=NO_THINKING), questions))

20 threads call it at once, but limiter lets only 5 run at the same time:

== 4. 并发 20 个请求，最多同时 5 个，每次调用记账
20 个请求用了 3.5 秒
第一条回答： HTTP 状态码 200 表示服务器成功处理了请求，并正常返回了所请求的资源。
日志共 20 条，总花费 0.000655 美元，第一条：{'time': '2026-09-14 21:35:48', 'model': 'deepseek-flash', 'seconds': 0.62, 'prompt_tokens': 16, 'completion_tokens': 19, 'cost_usd': 2.8e-05, 'attempts': 1}

The 20 requests took 3.5 seconds; one after another they'd take about 12. This run hit no errors that needed a retry, so every record's attempts is 1.

A few safeguards for your bill

Retries and concurrency control guard against accidents; these guard against the bill:

• Set max_tokens on every call. It stops the model from outputting endlessly. With thinking on, be generous; lesson 3 of module 00 showed that thinking uses this budget.
• Put a limit on every loop. Tool-calling loops and retry loops alike need a maximum count. That's why the tool-calling loop in lesson 3 stops after 5 rounds.
• Set balance alerts on the platform. DeepSeek is prepaid and stops when your balance runs out, so there's a natural ceiling. Still, it's best to top up only enough for a period of time, and keep an eye on the balance.
• Read the log. Glance at the total cost in calls.jsonl every day, and any unusual growth jumps out.

Exercises

1. Change limiter's concurrency to 1 and to 20, run each once, and compare the total time for the 20 requests.
2. Simulate a failure: define a function that raises openai.APITimeoutError on its first two calls and only really calls the model on the third. Swap it into call_llm and look at the retry and wait output, and at attempts in the log. (Hint: openai.APITimeoutError(request=...) needs a request argument; you can use httpx.Request("POST", "https://example.com").)
3. Write a small script that reads calls.jsonl and prints the total number of calls, total cost, average duration and the 3 slowest calls.

Self-check

1. Should you retry a 401 error? What about a 429?

A 401 means the key is wrong; retrying any number of times gives the same result, so report the error and have someone check the key. A 429 means too many requests and you've been rate-limited, which is temporary: wait a while and retry, with the wait growing each time.

2. Why should the wait between retries double each time, with a little randomness added?

Doubling gives the server time to recover; retrying often while the other side is overloaded only makes things worse. The randomness spreads out the retries of many requests that failed at the same time, so they don't all hit the server again at the same moment.

3. With nothing configured, does the openai SDK retry automatically?

Yes. The current version defaults to max_retries=2: on timeouts, connection failures, and status codes 408, 409, 429 and 500 and above, it waits a short time and retries automatically, up to 2 times. The default timeout is 600 seconds, which is too long for most interactive use, so set timeout yourself.