English
简体中文繁體中文English日本語DeutschFrançais
Module 06 · Building with LLMs

Evaluation and going live

Build an evaluation set, use a model as a judge and calibrate it, log every step, cut cost and latency, add input and output guardrails, and finally deploy RepoBot as a web service.

Lessons

  1. 01
    Building an evaluation set

    Prepare 36 questions for RepoBot covering docs questions, source questions, questions it should refuse, questions with no answer, and injections. Store them as JSONL, run them all with one command, tally steps, time and cost by category, then do a rough rule-based check.

    40 minutes · Intermediate
  2. 02
    Using a model as the judge

    Write scoring criteria for 36 answers, have a model judge them, then compare with human labels one by one. With the lessons of Module 04 applied, the judge agreed with the human labels on all 36, and caught an error hidden inside an explanation.

    45 minutes · Intermediate
  3. 03
    Logging and observability

    Write a tracer of a few dozen lines that records every model call and every tool call an agent makes as one line of JSON. Afterwards, from the logs alone, you can work out how much money and time each question took, which step was slowest, and which tool is failing.

    40 minutes · Intermediate
  4. 04
    Cutting cost and latency

    Five small experiments. Putting fixed material first saved 64%, asking for brevity cut output by 92%, simple questions go without thinking, a home-made result cache, and concurrency took 10 requests from 9.5 seconds to 2.3.

    35 minutes · Intermediate
  5. 05
    Guardrails: keeping out what shouldn't get in or out

    The input guardrail classifies questions with one cheap call; it got 41 of 42 right, and all of them after one added rule. The output guardrail masks keys and phone numbers in answers with regexes, and deals with secrets split in two during streaming.

    40 minutes · Intermediate
  6. 06
    Project: deploying the Q&A assistant

    Turn RepoBot into a web service with a streaming FastAPI endpoint, a streaming agent loop, input and output guardrails, trace logging and input validation, then deploy it to a server. This is where Part 1's running project is completed.

    90 minutes · Intermediate

An AI application that runs on your computer is still a long way from something you can hand to other people. You need to know whether it answers well, whether a change made it worse, whether you can find out why when something goes wrong, how much it will cost each month, and whether people will abuse it.

This module answers those questions. The first two lessons cover evaluation: preparing questions and using a model as the judge, then checking with human labels whether the judge can be trusted. Lesson 3 covers logging, Lesson 4 cutting cost and latency, Lesson 5 guardrails. The last lesson puts everything into RepoBot and turns it into a web service.

Why this order

Evaluation comes first, because every later change (optimising cost, adding guardrails, turning it into a web service) needs it to confirm nothing got worse. Logging comes before optimisation, because you need to know where the money and time go before you know what to optimise. Guardrails come last, because they block requests, and you need the evaluation set to confirm they don't block legitimate questions.

You're done when

  • You can prepare an evaluation set for your own application covering normal, edge, refusal and injection cases, and run it all with one command.
  • You can write the prompt for an LLM judge, and use a few dozen human labels to check how closely it agrees with human judgement.
  • You can add trace logging to an agent, and compute each request's steps, time and cost from the logs.
  • You can name at least three ways to save money or cut latency, and measure their effect on your own application.
  • You can implement an input classification guardrail and an output redaction guardrail, and know what each can't stop.
  • RepoBot v4 runs as a web service on your computer, and you can say what's still missing before it goes live.

Code for this module

Code and program output are shown exactly as they ran, so comments and printed output are in Chinese.