pyKT: a PyTorch benchmark harness for deep knowledge tracing

What pyKT is for, and who it is actually for

Knowledge tracing predicts whether a student will answer the next question correctly, given their history of attempts. Dozens of deep learning architectures now do this, and the published numbers are hard to compare because each paper preprocesses its own data, picks its own train/test split, and evaluates under a different scenario. pyKT exists to remove that source of noise. The README describes a standardized set of integrated data preprocessing procedures on more than 7 popular datasets across different domains, 5 detailed prediction scenarios, and more than 10 frequently compared DLKT approaches. The intended reader is someone writing a paper or a thesis who needs a baseline table that a reviewer will accept. The README does not describe an end-user API, a teacher dashboard, or a deployed service. Everything in the repository layout points at experiment running: configs/, data/, examples/ with files named wandb_akt_train.py and wandb_dimkt_train.py, and a multi_run_all.sh driver. If your goal is to embed a knowledge tracing model in a product, pyKT is a research harness you would have to strip down first.

How the training pipeline is put together

The architecture is a conventional experiment stack rather than a novel runtime. Datasets live under data/ and are converted by preprocessing code invoked from examples/data_preprocess.py and examples/dataprocess.sh. Model implementations sit in the pykt/ package, which setup.py installs via find_packages(). Training entry points are individual scripts under examples/, one per model, and nearly all of them carry a wandb_ prefix, which tells you the intended observability layer is Weights & Biases rather than stdout logging. That is a real design decision with consequences: wandb>=0.12.9 is a hard dependency in install_requires, so a run without a W&B account or an offline mode configured will need attention before the first script executes. Results are then post-processed by separate scripts, including examples/extract_raw_result.py, examples/generate_splitpred.py and examples/merge_wandb_results.py. The flow is therefore: preprocess once into a shared format, launch one or many training scripts, then merge the W&B artifacts into the comparison table. The README does not document the on-disk schema of the preprocessed tensors, so anyone extending preprocessing has to read the code.

Installing pyKT and running a first model

The README gives a conda-first installation. It pins Python 3.7.5, which is the single most important practical constraint, because that interpreter version is long past its upstream support window and will not be available in every environment.

conda create --name=pykt python=3.7.5
source activate pykt

The package itself is installed from PyPI. The README passes an explicit index URL.

pip install -U pykt-toolkit -i  https://pypi.python.org/simple 

After installation the declared dependencies are numpy>=1.17.2, pandas>=1.1.5, scikit-learn, torch>=1.7.0, wandb>=0.12.9 and entmax. Note that setup.py declares python_requires=">=3.5" while the README instructs Python 3.7.5, so the metadata is looser than the documented workflow. A first real run means picking a training script from examples/, for example examples/wandb_akt_train.py, and pointing it at a config from configs/. The README does not spell out the argument list for those scripts, so the docs site is the place to look before you type a command. Expect the script to initialize a W&B run and write metrics there, not to print a final accuracy to the terminal.

Where pyKT will disappoint you

The release history is the first warning. The recent releases list ends at v1.0.0, dated 2023-02-10, while setup.py still declares version 0.0.38. That mismatch means the version string inside the installed package may not match the tag you read about, and any tooling that reads pykt.__version__ will report the older number. Second, there is no documented inference or serving interface. Nothing in the README or the repository entries describes exporting a trained checkpoint, running a forward pass on a single new student interaction, or measuring latency. A team that wants online next-question prediction gets a training library and a research problem. Third, the W&B coupling makes offline or air-gapped reproduction harder than it looks, since the dependency is mandatory rather than optional. Fourth, the README does not document rollback, migration between versions, or how dataset splits changed between releases, so a number you reproduce today is not guaranteed to be comparable to one from an earlier tag. Finally, the hyperparameter tuning results are hosted as a Google Drive folder link, which is fine for reading but not for automated retrieval.

pyKT against EduKTM and single-model repositories

The README's own reference list points at the realistic alternatives. EduKTM is a general educational knowledge tracing model library from a different group, and its scope is broader than benchmark comparison: it targets knowledge tracing as a task family rather than a fixed set of DLKT baselines with shared splits. The second kind of alternative is the single-model repository, and the README lists plenty, including the AKT implementation, the GKT implementation and the SAKT PyTorch port. The difference in approach is stark. A single-model repo gives you one architecture, its authors' preprocessing, and its authors' evaluation, which is exactly the situation pyKT was built to escape. pyKT gives you many architectures behind one preprocessing pipeline and one scenario vocabulary, at the cost of adopting its environment, its W&B dependency and its data layout. If you only ever need one model and you trust its paper's numbers, the single-model repo is less machinery. If you need to argue that your method beats several baselines fairly, the shared pipeline is the point.

Licence, maintenance and upgrade cost

pyKT is MIT licensed, both in the LICENSE file at the repository root and in the classifiers block of setup.py. MIT is permissive: it allows commercial use and modification provided the copyright notice and permission notice are retained. That matters because the library reimplements or adapts model code from the third-party projects listed in the README, and those upstream repositories may carry their own licences. The pyKT LICENSE covers pyKT's own code; it does not automatically relicense anything vendored from elsewhere, so anyone shipping a derivative should check the corresponding upstream project. This is a factual observation about how the licence file works, not legal advice. On maintenance: the repository is not archived, and the last push was on 2026-08-31, so the codebase is still receiving commits even though the most recent tagged release is v1.0.0 from 2023-02-10. That gap between commit activity and release cadence is the upgrade cost in practice. Installing from PyPI gives you the packaged version; tracking new models means installing from the repository. The README does not describe a deprecation policy or a version compatibility matrix, so pinning a commit hash is the only reproducible option available.

Editorial conclusion

Adopt pyKT if you are a researcher or graduate student who needs to compare DLKT architectures on the same splits, and you accept a pinned Python 3.7.5 conda environment and a 2023 v1.0.0 release. Do not adopt it as the inference engine inside a live tutoring product: the README documents no serving path, no rollback and no latency figures. Before committing, check the docs at pykt-toolkit.readthedocs.io for the scenario definitions you intend to report, and confirm that the dataset you want is among the ones the preprocessing code covers.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. pykt-team/pykt-toolkit on GitHub
4. README
5. Releases