How to choose a model

New models arrive almost every week, and each one claims first place on some leaderboard. Choosing a model by leaderboard is like hiring on qualifications alone: it's worth something, but it can't tell you whether this person will do your particular job well.

This lesson covers a more reliable way: prepare 20 questions from your own task, have every candidate model answer them, and see which is right, which is fast and which is cheap. It doesn't cost much, and you have results in half an hour.

What to look at

Quality comes first, but it isn't everything. A model that's 95% accurate but makes users wait 10 seconds will feel slow in a chat window; one that's 90% accurate but ten times cheaper may suit a background job called hundreds of thousands of times a day better.

Rough kinds of model

Closed and open-weight. Closed models can only be called through an API; you never get the model itself. Open-weight models have their parameters published, so you can download them and run them on your own servers, keeping your data in-house, but you have to provide the GPUs and run them yourself. Module 10 covers deploying open-weight models locally.

Thinking and non-thinking. General models and reasoning models used to be two separate models; now many models switch between the two with a parameter. DeepSeek's deepseek-flash and deepseek-v4-pro both support thinking and non-thinking modes, with thinking on by default. Thinking mode is more accurate, but slower and more expensive.

Large and small. A provider's line-up usually has a cheap, fast model and a more expensive, more capable one, such as DeepSeek's flash and pro. Intuition says the expensive one must be better; the experiment below shows that it isn't necessarily.

As of September 2026, the main options you can use directly from mainland China include DeepSeek, Alibaba Cloud Model Studio (the Qwen series) and Kimi; lesson 2 of module 00 lists their API addresses. Specific models change quickly, so there's no list here; look up the latest in each provider's docs. What matters is the comparison method below, which won't go out of date.

Experiment: four configurations on 20 questions

I prepared 20 questions with right answers: arithmetic, unit conversion, counting letters, counting Chinese characters, simple reasoning, classical poetry and Python basics. Then I had four configurations answer them all: deepseek-flash and deepseek-v4-pro, each with thinking on and off.

The core of the program looks like this (the complete code is in code/01-llm-basics/model_eval.py; costs are worked out with the cost_usd written in lesson 4):

QUESTIONS = [
    ("17 乘以 23 等于多少？", ["391"]),
    ("9.11 和 9.9 哪个大？", ["9.9"]),
    ("“秋天的叶子一片片落下”这句话有几个字？", ["10"]),
    ("如果今天是星期三，100 天后是星期几？", ["星期五", "周五"]),
    ("鸡兔同笼，共 35 个头，94 只脚，兔子有几只？", ["12"]),
    # ……一共 20 道
]


def normalize(text):
    # 去掉空格和标点，再去掉结尾的单位和"大"这类多余的字，只比较核心答案。
    # 注意不能去掉小数点，否则 9.9 和 99 就分不清了。
    text = re.sub(r"[\s，。,!！？?：:“”\"'、]", "", text.lower()).rstrip(".")
    return re.sub(r"(个字|个|人|只|天|平方厘米|厘米|摄氏度|度|次|大)$", "", text)


def run_one(model, thinking, question):
    start = time.time()
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "只回答最终答案本身，不要写单位以外的任何解释。"},
            {"role": "user", "content": question},
        ],
        extra_body={"thinking": {"type": "enabled" if thinking else "disabled"}},
    )
    return response.choices[0].message.content.strip(), time.time() - start, response.usage

Each configuration answers the 20 questions using 10 concurrent threads, counting how many it gets right, the average seconds per question and the total cost, and printing the questions it got wrong. What I got:

deepseek-flash（不思考）: 答对 19/20，平均 0.8 秒/题，20 题共 0.0002 美元
    答错：9.11 和 9.9 哪个大？ → 9.11
deepseek-flash（思考）: 答对 20/20，平均 0.9 秒/题，20 题共 0.0026 美元
deepseek-v4-pro（不思考）: 答对 16/20，平均 0.9 秒/题，20 题共 0.0009 美元
    答错：9.11 和 9.9 哪个大？ → 9.11
    答错：“秋天的叶子一片片落下”这句话有几个字？ → 9
    答错：一个正方形的周长是 24 厘米，它的面积是多少平方厘米？ → 9
    答错：单词 raspberry 里有几个字母 r？ → 2
deepseek-v4-pro（思考）: 答对 20/20，平均 2.1 秒/题，20 题共 0.0114 美元

Reading the results

Thinking mode corrected every mistake. With thinking on, both models got all 20 right. The questions missed without thinking were all ones that need "one step at a time": comparing decimals, counting characters, working out a side length before the area.

The expensive model isn't necessarily better. With thinking off, deepseek-v4-pro got only 16 right, 3 fewer than the cheaper flash. On the square question it answered "9", probably taking "perimeter 24" as a clue to the area. More expensive models usually have the edge on harder, more open-ended tasks, but on your particular task the cheaper one can easily beat them. That's exactly why you have to test for yourself.

Equally perfect, very different in price and speed. Flash with thinking and pro with thinking both scored 20/20, but pro cost more than 4 times as much (US$0.0114 against 0.0026) and was more than twice as slow (2.1 against 0.9 seconds). For this set of questions, flash with thinking is clearly the better choice.

Flash without thinking gives the best value. 19/20 accuracy, at a tenth of the cost with thinking on. If your task has no traps like "comparing decimals", it may be all you need.

The limits of this experiment

Having given the conclusions, I should be just as clear about where this experiment can't be trusted.

20 questions are too few. The difference between 16 and 17 right may be just luck. I ran it twice, and deepseek-v4-pro without thinking scored 17 one time and 16 the other, missing not quite the same questions. Twenty questions are good for spotting "clearly much worse"; to tell apart two models that are "about the same", you need more questions, and several runs of each.

The grading script makes mistakes too. On my first run, flash with thinking was marked wrong on two questions: one answer was "9.9 大" (9.9 is bigger) and one was "10个字" (10 characters). The answers were actually right; my normalize function just didn't strip tails like "大" and "个字". Only after fixing it and rerunning did I get the results above. So always go through the questions marked wrong one by one, and make sure it was the model that erred and not the grading.

These questions aren't your task. Arithmetic, counting letters and classical poetry have nothing in common with customer service, writing code or extracting contract details. This set only demonstrates the method. To choose a model for your own project, the questions must come from your real task.

Choosing a model for your own project

1. Collect 20 real questions. Take them from your real use: questions users actually asked, documents you actually need to process. Include easy ones, and ones you know models tend to get wrong.
2. Decide how to judge right and wrong. Questions with a right answer are best, compared automatically by a program. For open-ended answers (say, writing a customer-service reply), write a scoring guide and grade by hand, or have another model grade them; lesson 2 of module 06 covers how.
3. Pick 2–4 candidates. Usually a cheap one and an expensive one, plus thinking on and off.
4. Run it and look at the mistakes. Don't just look at accuracy; look at every wrong answer and decide whether the fault is the model's or the question's or the grading's.
5. Price it at your real volume. Use lesson 4's cost_usd to estimate the monthly bill from how many calls you make a day.
6. Choose the cheapest one that meets your requirements. Once accuracy is good enough, look at speed and price.

Don't throw these 20 questions away. Whenever you switch models, change a prompt or upgrade a version, run them again and you'll know whether anything got worse. Module 06 expands them into a full evaluation set.

Exercises

1. Add 5 questions you think models are likely to get wrong to QUESTIONS in model_eval.py, such as multi-step date arithmetic or logic puzzles with a trap, and see which of the four configurations does best.
2. If you have an API key from another provider, add their models to CONFIGS (you'll need to change run_one so it can use different clients, and drop the thinking parameter that other providers don't recognize), and compare the results.
3. Run model_eval.py 3 times in a row on the same configuration. Is the number right the same each time? Which questions are sometimes right and sometimes wrong?

Self-check

1. A model is first on a leaderboard. Why test it with your own questions anyway?

Leaderboards measure a model's average level on general questions, while you care about how it does on your particular task, and the two can differ a lot. In this lesson's experiment, the more expensive deepseek-v4-pro made more mistakes without thinking than the cheaper flash. Only testing on real questions from your own task tells you which model suits you.

2. On 20 questions, model A gets 17 right and model B gets 16. Does that show A is better than B?

No. Twenty questions are too few; a difference of one or two may be random variation, and the same model can score differently on two runs. A small test set is good for spotting clear gaps. To tell close models apart you need more questions, with several runs of each averaged.

3. Your evaluation script says the model got 3 questions wrong. What do you do next?

Look at the raw answer to each of the 3 questions and check whether the model really got it wrong or the grading script marked it wrong (for example because the answer had an extra unit or punctuation mark). Also check the questions and reference answers themselves. Once that's confirmed, decide from the kind of mistakes whether to switch models, turn thinking on or change the prompt.