pyKT: a benchmarking harness for deep learning knowledge tracing models
pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models
At a glance
- What is it?
- pyKT packages seven or more public datasets, five prediction scenarios and over ten DLKT model implementations behind one PyTorch training loop. It is a benchmark runner for researchers, not a student-facing tutoring component, and its value depends on whether its data splits match the question you are asking.
- Who is it for?
- Adopt pyKT if you need to compare a new knowledge tracing architecture against DKT, SAKT, AKT and the rest under one preprocessing pipeline and one set of prediction scenarios, because rebuilding that comparison by hand is the expensive part. Do not adopt it if you need a production student model with a serving path and online updates; nothing in the README suggests pyKT provides either.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 15 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The comparison problem pyKT was built to remove
Knowledge tracing papers report numbers on the same handful of datasets, but the numbers are rarely comparable. Each group writes its own preprocessing, its own train/validation/test split, and its own definition of what counts as a prediction. pyKT's stated purpose is to fix that by putting the preprocessing, the scenario definitions and the model implementations in one library. The README describes a standardized set of integrated data preprocessing procedures on more than 7 popular datasets across different domains, 5 detailed prediction scenarios, and more than 10 frequently compared DLKT approaches. The intended user is a researcher who has a new architecture and needs a defensible baseline table, or a reviewer who wants to reproduce one. It is not aimed at a team that wants to ship a knowledge tracing model inside a learning product. The library stops at training and evaluation; the README describes no serving layer, no feature store integration and no online update path.
What the repository actually contains
The README gives the shape of the library in one sentence: it is built upon PyTorch, and it bundles preprocessing, scenarios and models. It does not document the directory layout, so the internal module names are not something I can state. What the README does make concrete is the reference list. Eighteen external repositories are cited as the sources the implementations were drawn from, including knowledge-tracing-collection-pytorch, AKT, GKT, HawkesKT and DTransformer. That list matters more than it first appears: it tells you pyKT is largely a consolidation effort, so a bug in a model may originate upstream, and a fix may need to be ported rather than written. The paper list runs to thirty-plus entries, from DKT through DKT+, DKVMN, SAKT, SAINT, AKT, simpleKT, SparseKT, DTransformer and on to recent work like MoCKT and DenoiseKT. The gap between that list and the stated count of more than 10 integrated approaches is the single most important thing to check before adopting. Some of those papers are listed as references, not as shipped implementations. The README does not label which is which.
Installation and the Python version constraint
The README gives a two-step install. First create the environment:
conda create --name=pykt python=3.7.5 source activate pykt
Then install from PyPI:
pip install -U pykt-toolkit -i https://pypi.python.org/simple
The pinned interpreter version is the detail worth pausing on. Python 3.7.5 is the version the README specifies, and that release is past end of life. A fresh environment built exactly as documented will pull an old interpreter and an old PyTorch stack alongside it. Whether newer Python versions work is not stated in the material I have, so treat the 3.7.5 line as the documented configuration and the newer one as untested until you check it yourself. The explicit -i flag points pip at the plain PyPI index rather than a mirror, which is a small thing but it means the command will behave the same regardless of any local pip.conf.
Scenarios are the part that decides whether your numbers mean anything
The five prediction scenarios are the most consequential design choice in the library and the least explained in the README. Knowledge tracing evaluation is not one task. Predicting the next interaction for a student already in the training data is a different problem from predicting the first interactions of a student the model has never seen, and the two produce very different accuracy figures on the same model. By fixing five scenarios and applying them uniformly, pyKT makes cross-paper comparison possible, but it also means the scenario you pick determines your headline number more than your architecture does. The README states the count and does not describe the five. If you are reporting results, read the scenario code before you read your model's output. The hyperparameter tuning results are published separately, at a Google Drive link given in the README, covering all DLKT models on the various datasets. That is where the reference numbers live, and it is a Drive folder rather than something in the repository, which means the benchmark record and the code have separate lifetimes.
Where pyKT is the wrong tool
Three cases. First, if you need a deployed model. pyKT trains and evaluates; the README describes no inference server, no batch scoring job and no way to update a fitted model as new interaction logs arrive. A team that needs next-week predictions for live students will have to build that layer themselves, and the model weights pyKT produces are only part of it. Second, if your dataset is not one of the bundled ones. The preprocessing is described as standardized and integrated for the supported datasets, which is a benefit when you use them and a cost when you do not, because you inherit a schema you did not choose. Third, if your question is about a single model rather than a comparison. Pulling in a full benchmarking library to train one DKT variant adds a dependency and a preprocessing contract for no gain; the upstream repositories cited in the README are the more direct route. There is also a maintenance consideration. The most recent release listed is v1.0.0 from February 2023, with v0.0.38 and v0.0.37 before it in 2022. The repository has seen pushes since then, but the tagged releases are old, so if you pin to a release you are pinning to a 2023 snapshot of a field that has moved.
How it differs from EduKTM and the upstream repositories
The closest comparison in the material is EduKTM, cited in the README's project list. Both are Python knowledge tracing libraries, and the difference is in what they optimise for. pyKT's framing is benchmarking: a fixed set of datasets, a fixed set of scenarios, and models scored against each other under one pipeline. That design is what makes its numbers comparable and also what makes it rigid, because the pipeline is the point. The individual upstream repositories, such as AKT, GKT or DTransformer, take the opposite approach: each ships one model with its own training script and its own data handling, which is more flexible per model and useless for a like-for-like table. If your goal is a leaderboard, pyKT's consolidation is the whole value. If your goal is to modify one model's internals, the upstream repository will be easier to work in, because you are not routing your change through a shared abstraction that was written to keep ten other models comparable.
Licence and the cost of staying current
pyKT is MIT licensed. That is permissive: it allows use, modification and redistribution with the licence and copyright notice retained, and it carries no copyleft obligation on your own code. It also means no warranty, which is standard and worth remembering given that the library reimplements models from eighteen external repositories, each of which may carry its own licence terms. The README does not state the licence of those upstream projects, so if you redistribute pyKT or a derivative, check the provenance of the specific model you are shipping rather than assuming MIT covers everything in the tree. On maintenance: the release cadence visible in the material is three tagged releases between July 2022 and February 2023, with repository activity continuing afterward. The practical cost is not the upgrade itself but the re-benchmarking. If a new release changes preprocessing or a scenario definition, your previously reported numbers are no longer comparable to the new ones, and you will need to rerun the baselines. Budget for that, and pin your dependency version in whatever environment file you use so a silent upgrade does not invalidate a results table.
Editorial conclusion
Adopt pyKT if you need to compare a new knowledge tracing architecture against DKT, SAKT, AKT and the rest under one preprocessing pipeline and one set of prediction scenarios, because rebuilding that comparison by hand is the expensive part. Do not adopt it if you need a production student model with a serving path and online updates; nothing in the README suggests pyKT provides either. Before you commit, verify two things: that the dataset split and the prediction scenario you intend to report match what the bundled preprocessing produces, and that the model you care about is present in the current tree rather than only in the paper list, since the README's paper list is longer than its stated count of integrated approaches.
Community notes