Temperature and sampling: why the same question gets different answers

Ask a large language model the same question twice and you'll often get two different answers. For writing copy that's a good thing: you can ask for a few versions and pick one. But if you're building a program that "extracts the name of Party A from a contract", and today it extracts "某某有限公司" (So-and-so Co., Ltd.) while tomorrow it's "某某公司" (So-and-so Company), that's a real nuisance.

The knob that controls this is called temperature. This lesson first works out where the randomness comes from, then implements temperature in code by hand, and finally looks at its effect on real API calls.

Where the randomness comes from

As the last lesson showed, at every step the model produces a probability distribution: how likely the next token is to be "爬山" (hiking), how likely "图书馆" (the library), and so on. Then a program called the sampler draws one token according to that distribution. The model itself computes almost the same distribution every time; the randomness comes from the drawing.

You can see the distribution directly. Ask deepseek-flash to continue "周末我打算去" (this weekend I'm planning to go…) with the logprobs parameter on (the last lesson showed how), and the top 10 candidates it returns for the first token are:

The most common approach isn't to always pick the most likely candidate, but to draw by probability: "爬山" 68.8% of the time, "图书馆" 14.9% of the time, and occasionally even "露营".

In my experiments I hit a case that shows this nicely. Asked to continue "我今天中午吃了" (for lunch today I had…), the most likely first token was "一碗" (a bowl of, 70.8%), followed by "我今天" (today I, 27.8%). That time it happened to draw the runner-up, and the whole answer turned into a restatement of the original sentence. One step drew a poor token, every later step built on it, and the answer went off in another direction. That's also why, when one answer comes out poorly, simply regenerating it often fixes things.

What temperature does

Temperature squeezes the distribution sharper or spreads it flatter before the draw.

The algorithm: take the log of each candidate's probability, divide by the temperature, and turn the results back into probabilities (a step called softmax). As a formula, new_p_i ∝ exp(log(p_i) / T), which is the same as raising p_i to the power 1/T and renormalizing.

• Temperature below 1: large probabilities get larger and small ones smaller; the distribution sharpens and results become more stable.
• Temperature 1: the distribution is unchanged.
• Temperature above 1: the distribution flattens, lower-ranked candidates get more chances, and results become more varied, and more likely to go off track.
• Temperature approaching 0: always pick the most likely candidate, which is called greedy decoding.

Using the real distribution above, let's do it by hand:

import numpy as np

candidates = ["爬山", "图书馆", "公园", "山里", "郊", "超市", "逛街", "逛", "书店", "露营"]
probs = np.array([0.6879, 0.1489, 0.0797, 0.0244, 0.0158, 0.0157, 0.0107, 0.0062, 0.0029, 0.0013])
probs = probs / probs.sum()  # 只取了前 10 个，重新归一化让它们加起来等于 1


def apply_temperature(p, t):
    # 温度作用在对数概率上：先取对数，除以温度，再变回概率（softmax）
    logits = np.log(p) / t
    e = np.exp(logits - logits.max())  # 减去最大值是为了防止 exp 溢出，不影响结果
    return e / e.sum()


def show(title, p, n=1000, seed=0):
    rng = np.random.default_rng(seed)
    draws = rng.choice(len(p), size=n, p=p)
    counts = np.bincount(draws, minlength=len(p))
    print(title)
    for word, prob, count in zip(candidates, p, counts):
        if prob > 0.0005 or count:
            print(f"  {word:　<4} 概率 {prob:6.1%}  抽中 {count:4d} 次")
    print()


show("原始分布（温度 1.0），抽 1000 次：", probs)
show("温度 0.5：", apply_temperature(probs, 0.5))
show("温度 1.5：", apply_temperature(probs, 1.5))

The output:

原始分布（温度 1.0），抽 1000 次：
  爬山　　 概率  69.2%  抽中  673 次
  图书馆　 概率  15.0%  抽中  166 次
  公园　　 概率   8.0%  抽中   83 次
  山里　　 概率   2.5%  抽中   29 次
  郊　　　 概率   1.6%  抽中   11 次
  超市　　 概率   1.6%  抽中   15 次
  逛街　　 概率   1.1%  抽中   12 次
  逛　　　 概率   0.6%  抽中    8 次
  书店　　 概率   0.3%  抽中    2 次
  露营　　 概率   0.1%  抽中    1 次

温度 0.5：
  爬山　　 概率  94.1%  抽中  944 次
  图书馆　 概率   4.4%  抽中   41 次
  公园　　 概率   1.3%  抽中   14 次
  山里　　 概率   0.1%  抽中    0 次
  超市　　 概率   0.0%  抽中    1 次

温度 1.5：
  爬山　　 概率  49.6%  抽中  467 次
  图书馆　 概率  17.9%  抽中  188 次
  公园　　 概率  11.8%  抽中  127 次
  山里　　 概率   5.4%  抽中   64 次
  郊　　　 概率   4.0%  抽中   40 次
  超市　　 概率   4.0%  抽中   40 次
  逛街　　 概率   3.1%  抽中   30 次
  逛　　　 概率   2.1%  抽中   21 次
  书店　　 概率   1.3%  抽中   16 次
  露营　　 概率   0.8%  抽中    7 次

At temperature 0.5, "爬山" rises from 69% to 94%, and the candidates behind it hardly get a look-in. At temperature 1.5, "爬山" drops to about half, and "露营" goes from 1 draw in 1,000 to 7.

Note that this is only the first token. An answer has dozens or hundreds of tokens, and every step draws like this. The differences at each step add up, so at high temperature whole answers differ enormously.

top_p: cutting off the long tail

The other common parameter is top_p, also called nucleus sampling. It doesn't change the relative sizes of the probabilities. Instead it sorts the candidates from most to least likely, adds them up from the top, stops once the total reaches top_p, throws away everything after that, and draws from what's left.

def top_p_filter(p, top_p):
    order = np.argsort(p)[::-1]
    cumulative = np.cumsum(p[order])
    # 保留累计概率刚好达到 top_p 的那几个，其余的概率清零
    keep = order[: np.searchsorted(cumulative, top_p) + 1]
    q = np.zeros_like(p)
    q[keep] = p[keep]
    return q / q.sum()


show("温度 1.0 + top_p 0.9：", top_p_filter(probs, 0.9))

温度 1.0 + top_p 0.9：
  爬山　　 概率  75.1%  抽中  733 次
  图书馆　 概率  16.2%  抽中  183 次
  公园　　 概率   8.7%  抽中   84 次

The first three candidates add up to just over 90%, so the draw happens among them only, and "露营", "超市" and the like never appear. The benefit is that it blocks tokens that are very unlikely but would send the answer off track if they were drawn.

Usually you only need to adjust one of temperature and top_p; adjust both and it's hard to tell which one did what.

Measuring it through the API

That was a local simulation. What happens with real calls? I asked deepseek-flash to write a metaphor about autumn, 20 times at each of three temperatures:

import os
from collections import Counter
from concurrent.futures import ThreadPoolExecutor

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LLM_API_KEY"],
    base_url=os.environ.get("LLM_BASE_URL", "https://api.deepseek.com"),
)
MODEL = os.environ.get("LLM_MODEL", "deepseek-flash")
QUESTION = "用一句话写一个关于秋天的比喻，不超过十五个字，只输出这句话。"


def ask(temperature):
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": QUESTION}],
        temperature=temperature,
        # 思考模式下 temperature 不起作用，所以要关掉思考
        extra_body={"thinking": {"type": "disabled"}},
    )
    return response.choices[0].message.content.strip()


for t in [0, 0.7, 1.3]:
    # 10 个线程同时发请求，省点时间
    with ThreadPoolExecutor(10) as pool:
        answers = list(pool.map(ask, [t] * 20))
    counts = Counter(answers)
    print(f"temperature={t}: 20 次里有 {len(counts)} 种")
    for text, n in counts.most_common(5):
        print(f"   {n:2d}× {text}")

What I got (yours will differ):

temperature=0: 20 次里有 2 种
   11× 秋天像一封写满离别的信。
    9× 秋天像一封缓缓飘落的信。
temperature=0.7: 20 次里有 6 种
   10× 秋天像一封写满离别的信。
    4× 秋天像一封慢慢变黄的信。
    3× 秋天像一封缓缓飘落的信。
    1× 秋天像一封缓缓拆开的旧信。
    1× 秋天像一封缓缓展开的旧信。
temperature=1.3: 20 次里有 15 种
    4× 秋天像一封缓缓飘落的信。
    2× 秋天像一封写满离别的信。
    2× 秋天像一封慢慢变黄的信。
    1× 秋天像一封写给大地的金色信笺。
    1× 秋天是把金色小提琴，风一拉就落叶。

(Only the 5 most common answers at each temperature are shown.)

Going from temperature 0 to 1.3, the number of different answers out of 20 rose from 2 to 15. At 1.3, a line like "秋天是把金色小提琴，风一拉就落叶" (autumn is a golden violin; when the wind draws the bow, leaves fall) appeared, unlike anything before it.

Two things are worth pointing out.

Even at temperature 0, the result isn't the same every time. In theory temperature 0 should always pick the most likely token, making the result fully deterministic. In practice two different answers appeared in 20 tries. The reason is on the server side: your request is computed in a batch together with other people's, and floating-point operations on the GPU happen in different orders, producing tiny differences in the results. Usually that doesn't matter, but when two candidates are extremely close (which is probably the case for "写满离别的" and "缓缓飘落的" here), that tiny difference is enough to flip the choice. Once one step picks a different token, everything after it differs. So don't expect temperature 0 to make results 100% reproducible. A program that needs stable output should validate it in code, not bet on the model being the same every time.

When the model is very sure, raising the temperature doesn't help. I also tried asking the model to "name a small shop that sells hand-made coffee". At temperature 1.5 I asked 5 times and got "豆语咖啡" (Bean Talk Coffee) all 5 times. My guess is that the first token, "豆语", was already so close to 100% likely that even a flattened distribution left it far ahead. If you want varied results, rather than raising the temperature, ask for them in the prompt: "give 10 names in different styles".

Temperature has no effect in thinking mode

DeepSeek's documentation states that in thinking mode the temperature parameter has no effect (setting it doesn't raise an error; it's simply ignored), and any top_p value below 0.95 is automatically raised to 0.95. That's why all the code above turns thinking off.

So if you want to control output with temperature, you have to turn thinking mode off. Conversely, with thinking on, don't expect adjusting the temperature to make output more stable.

What value to use

DeepSeek's official recommendations (as of September 2026):

DeepSeek's default temperature is 1.0. This table applies only to DeepSeek; other providers' models react to temperature differently, and the same number can have quite a different effect.

My approach: for tasks with a right answer, such as extraction, classification and reformatting, set temperature to 0; for writing, naming and brainstorming, use the default or go higher; if you're unsure, start with the default and adjust if you're not happy.

Exercises

1. Change the temperature in sampling.py to 0.1 and 3.0 and see what the samples look like. At temperature 3.0, how often does "露营" get drawn?
2. Change top_p to 0.7 and 0.99 and see how many candidates are kept.
3. Use temperature_api.py with a different question: have the model translate "The quick brown fox jumps over the lazy dog" into Chinese, 20 times at temperature 0 and 20 times at 1.3, and count the distinct translations. Does temperature matter more or less for translation than for writing metaphors? Think about why.
4. Use last lesson's logprobs parameter to look at the first token's candidates and probabilities when the model is asked to "name a small shop that sells hand-made coffee". Does it confirm my guess about the "豆语咖啡" effect?

Self-check

1. Why does lowering the temperature make the model's answers more stable?

Temperature acts on the candidate probability distribution at each step: the lower it is, the higher the likely candidates go and the lower the unlikely ones, making the distribution "sharper". The draw almost always lands on the same token, so answers naturally become more stable. As temperature approaches 0, every step simply picks the most likely candidate.

2. You set the temperature to 0, ask the same question 20 times, and still get two different answers. Is that normal?

Yes. When the server processes requests in batches, the order of floating-point operations changes, which causes tiny differences in the probabilities. When two candidates are extremely close, that difference is enough to flip the choice, and everything after it differs. So temperature 0 can't be relied on for fully reproducible results.

3. On DeepSeek with thinking mode on, you lower the temperature from 1.0 to 0 and the output doesn't become more stable. Why?

DeepSeek ignores the temperature parameter in thinking mode, so setting it has no effect. To control output with temperature, turn thinking off with extra_body={"thinking": {"type": "disabled"}}.