The Little Book of ML Metrics: A NannyML Reference Handbook Built in Jupyter Notebook

What a Metrics Handbook Solves That a Library Does Not

Metric implementations are easy to find and hard to choose between. scikit-learn ships dozens of scoring functions, and each one has a signature and a default behaviour but no opinion about when you should reach for it. The Little Book of ML Metrics targets the gap between knowing a function exists and knowing which one answers your question. The README frames the intent directly: the book is meant to be "this little handbook that sits on every data scientist's desk for quick reference," spanning from accuracy to what the authors call the most obscure metrics, naming the P4 metric as an example. The stated audience is anyone from an aspiring data scientist to an experienced professional, and the authors argue the topic is underserved because metrics are "often overlooked in traditional data science courses and university degrees." That is a reasonable claim about curricula. The repository itself is the book's source, written in Jupyter Notebook, with figure assets stored under book/figures. If you want a formula, an intuition and a short discussion of trade-offs in one place, that is the format. If you want a callable function, this is the wrong repository.

How the Book Is Structured and Built

The primary language of the repository is Jupyter Notebook, which tells you how the content is authored: metric explanations live in notebooks, and the figures referenced by the README (for example book/figures/The_Little_Book_of_Metrics_MAPE.png) are stored as image files alongside them. The README does not describe the build pipeline in detail, so the exact toolchain that converts notebooks into the PDF is not confirmed by the supplied material. What is confirmed is the output path: a nightly pre-release PDF is published as a release asset, and the README links to it at releases/download/nightly/main.pdf. The single release listed is tagged nightly, dated 2024-10-14, described as a Nightly build. That naming implies a recurring automated build rather than versioned editions, though the repository metadata shows only one such release, so the cadence cannot be verified from what is here. The content is organised by metric family rather than by model type: regression, classification, clustering, ranking, computer vision, NLP, GenAI, probabilistic, bias and fairness, and data observability. Ten categories is broad coverage for a single volume, and breadth at that scale usually means each metric gets a compact treatment rather than a chapter-length one. Treat the table of contents as the contract: it tells you what is in scope, and the compactness is the trade you accept for having everything in one PDF.

Getting the PDF and Working With the Source

There is no package to install. The README's practical instruction is the preview link, and the download URL follows a predictable pattern: github.com/NannyML/The-Little-Book-of-ML-Metrics/releases/download/nightly/main.pdf. Fetching that file with curl or opening it in a browser is the whole setup for a reader. For contributors, the workflow is different: clone the repository, work in the notebooks, and follow the guidance in CONTRIBUTING.md, which the README points to for details. The README does not list environment files, dependency manifests or build commands, so anyone planning to render the PDF locally will need to inspect the repository tree for those, and I cannot confirm from the supplied material which tools are required. The print edition is sold through nannyml.com/metrics, and the README states that revenue from printed copies supports further development and maintenance. There is an unusual incentive layer on top: contributors receive an affiliate link with 10 percent commission on sales through it and a credit in the book, while reviewers who apply through the linked Google Form receive 15 percent and the same credit. That is a transparent arrangement, and it is worth knowing before you volunteer, because your name in the book is tied to a commercial funnel rather than to a purely academic contribution record.

Where the Repository Falls Short as a Reference

The licence is the first problem. Repository metadata lists the license as unknown, and the README does not name one either. For a book, that is not a cosmetic gap. Text and figures are creative content, and without an explicit licence you cannot assume the same freedoms you would get from an MIT-licensed code library. The README says the book is open source and that the digital version is freely accessible, which describes access, not reuse rights. If you want to reproduce a figure in a course, translate a section, or bundle the PDF internally, the safe reading is that permission is not granted by default. Ask the maintainers. The second limitation is the release channel. A nightly tag means the artefact you download today may differ from the one you read last month, and there is no numbered edition to cite. In a teaching context, that makes stable references awkward: you can cite the book, but not a fixed version of it. The third is depth. Ten metric categories in one volume means the GenAI and data observability sections cannot carry the same weight as a dedicated text on either subject. The README makes no claim otherwise, but a reader who arrives expecting a definitive treatment of fairness metrics will find an introduction, not a survey of the literature.

How It Compares With a Metrics Library Like scikit-learn

The obvious alternative is not another book. It is scikit-learn's metrics module, or a similar library, which gives you functions you can call in a pipeline. The difference is in what each artefact is for. scikit-learn's metrics answer "compute this number for these arrays" and leave the interpretation to you; its documentation pages are reference material tied to an API surface. The Little Book of ML Metrics inverts that. It is organised around explanation and selection, with no importable interface described in the README and no installation instructions. You cannot call it from a notebook. What you can do is read it while deciding whether your ranking problem wants NDCG or MRR, then go to a library to compute the one you chose. The two are complementary, and the repository does not pretend to replace a library. It is also worth noting the publisher's context: NannyML is described in the README as an open source Python library for ML monitoring and post-deployment data science, and the book's inclusion of a data observability category reflects that interest. That is relevant if you work on monitoring, and less relevant if you do not.

Maintenance, Contributions and the Printing Business

The repository was last pushed on 2026-09-08, which suggests active development at the time of writing, though the only release listed is the nightly build from 2024-10-14. Those two facts sit oddly together, and the discrepancy is not explained in the supplied material. It may mean the nightly release asset is regenerated in place rather than re-tagged, but that is inference, not a documented policy. For a reader, the practical consequence is that you should check the release page rather than assume the PDF matches the timestamp of the last commit. Maintenance has a funding model attached. The README states that printed copies generate revenue that supports development, and that contributors, reviewers and authors share in sales through affiliate links. That is a more explicit arrangement than most documentation projects, and it shapes what the project optimises for: a polished, saleable print edition rather than a fast-moving software artefact. If you contribute, expect editorial review rather than a merge-and-move-on code review. If you cite it, cite the book as a whole, because there is no versioned edition to pin.

Who Should Keep This on the Desk

The book fits a specific reader: someone who already knows the metric families and wants a compact second opinion, or a practitioner moving into an unfamiliar area such as ranking or fairness and needing orientation before opening a library. It also fits teams that want a shared vocabulary document, provided they verify the licence before circulating copies. It does not fit anyone who needs executable code, a versioned citation, or an authoritative deep treatment of GenAI evaluation. The authors' own framing supports that boundary: the book is a quick reference, not a textbook. Before adopting it, check three things in order. Open the nightly PDF and confirm your metric is covered at the depth you need. Check the repository for a licence file, since the metadata field is empty. And if you plan to contribute, read CONTRIBUTING.md and decide whether the affiliate arrangement described in the README matches what you expect from an open source contribution. Those three checks take a few minutes and settle whether this handbook belongs on your desk or merely in a browser tab.

Editorial conclusion

Adopt it as a desk reference if you already know which metric family you need and want a second explanation of it; skip it if you need executable metric implementations, since the repository is a book written in Jupyter Notebooks and the README points readers to the PDF rather than to an installable package. Before relying on it, verify two things: the licence, which the repository metadata leaves blank and which matters if you plan to reuse figures or text, and whether the nightly PDF at the releases URL is current, since the only listed release is the nightly build from October 2024. The book covers regression, classification, clustering, ranking, computer vision, NLP, GenAI, probabilistic, bias and fairness, and data observability metrics, so confirm your specific metric appears in the table of contents before treating it as your reference.

Official sources

1. Issues
2. NannyML/The-Little-Book-of-ML-Metrics on GitHub
3. Project website
4. README
5. Releases