TensorKart: End-to-End Imitation Learning on Mario Kart 64
self-driving MarioKart with TensorFlow
At a glance
- What is it?
- TensorKart is a Python project that records joystick input alongside emulator screenshots and trains a TensorFlow model to reproduce that driving. It is a teaching-sized end-to-end imitation learning pipeline, not a general game AI framework, and its value depends on how much demonstration data you are willing to record by hand.
- Who is it for?
- Adopt TensorKart if you want a small, readable end-to-end imitation learning example that runs against a real emulator and you accept that every training sample comes from your own manual driving. Do not adopt it if you need a headless, reproducible benchmark or a framework that supports multiple games; the play loop depends on gym-mupen64plus, and the README's own notes warn that some captured screenshots can be the desktop instead of the emulator.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 43 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem TensorKart actually solves
TensorKart targets a narrow but concrete task: teaching a model to drive Mario Kart 64 by copying a human. The README frames the workflow around recording a joystick and the matching screen frames, then training a network to map one to the other. The training set described there is small and hand-made: 4 races on Luigi Raceway, 2 races on Kalimari Desert, and 2 races on Mario Raceway. That is the whole corpus. The README states that with even a small training set the model is sometimes able to generalize to a new track, and it shows Royal Raceway as the untrained example.
The intended user is someone who wants to see an end-to-end pipeline in full: capture, label by driving, train, evaluate in the emulator. It is not aimed at teams who need a driving benchmark with fixed datasets or leaderboards. The only labels are the ones you create with a controller, so the project's ceiling is set by how patiently you drive.
Screenshots in, joystick vectors out
The data flow is visible in the README's description of the training matrices. Preparing data with `python utils.py prepare samples/*` builds an `X` and a `y` pair. `X` is described as a 3-Dimensional array of images. `y` is the expected joystick output as an array with five positions: joystick x axis, joystick y axis, button a, button b, and button rb. That is a regression target, not a classification over discrete actions, so the model is being asked to predict continuous stick positions and three button states from pixels.
At play time the flow reverses. The README says play.py uses the gym-mupen64plus environment, which supplies screenshots of the emulator; those images go to the model, and the returned joystick command is sent to the environment. A human can take over by holding the LB button on the controller, which overrides the AI commands. That override is the practical safety valve: without it, a bad prediction would be applied with no way to intervene until the race ends.
The architecture itself is not specified in the supplied material. The README credits SullyChen/Autopilot-TensorFlow under special thanks, which suggests the network is in that lineage, but the exact layer stack cannot be confirmed from what is given here. Treat the model shape as something you read from train.py rather than something the README promises.
Getting it running: the commands the README gives
Dependencies are split between Python and system packages. The README says to install `python` and `pip`, then run `pip install -r requirements.txt`, and to install `mupen64plus` via apt-get. There is no container, no lockfile mentioned, and no pinned version list in the material, so reproducing an old TensorFlow environment is left to the reader.
Recording has a physical precondition that is easy to miss: a joystick must be connected and mupen64plus must be using the sdl input plugin. You start the emulator, run `record.py`, confirm the graph responds to joystick input, and position the emulator window so the program captures it from the top left corner. Then you press record and drive. The README notes the GUI stops updating while recording to avoid slowdowns, and it warns to double check samples because sometimes the screenshot is the desktop instead. Those bad rows are removed by deleting lines from `data.csv`.
Inspection and preparation come next: `python utils.py viewer samples/luigi_raceway` opens a sample directory, and `python utils.py prepare samples/*` builds the matrices. Training is `train.py`, which the README says uses TensorFlow with cuDNN for GPU acceleration and can take about an hour depending on data volume and system specs. It saves the best model from all epochs to disk when finished. Playback is `play.py`.
Where the pipeline breaks down
The most serious limitation is stated in the README itself: some captured screenshots are the desktop rather than the game. Because the capture is tied to window position rather than to the emulator's own framebuffer, any window overlap, alt-tab or resize silently injects garbage into `X` while `y` still holds a plausible joystick value. The suggested fix is manual line deletion in `data.csv`, which does not scale and is easy to do incompletely.
There is a second, quieter failure mode. The model is trained on frames from a fixed capture region and a fixed set of tracks, then evaluated on a track it has never seen. The README presents Royal Raceway generalization as a possibility (the model is sometimes able to generalize), not a guarantee. Off-distribution frames have no correction mechanism beyond the LB override, and the reward signal mentioned under future work, `-1` per time-step, is described as a metric for later reinforcement work rather than something the current training loop optimizes.
Finally, the project is the wrong tool if you need a headless, deterministic benchmark. Recording requires an interactive desktop session with a visible emulator window, so the data collection step cannot be scripted into a CI job as described.
How it differs from Donkey Gym and SerpentAI
The README lists related projects, and the contrast is instructive. Donkey Gym provides OpenAI Gym environments for the Donkey Car, a physical RC vehicle; TensorKart instead drives a Nintendo 64 emulator through gym-mupen64plus. The difference is not just the platform. Donkey Car projects typically ship a defined car platform and a simulation path, while TensorKart's environment is the emulator you already have running on your desktop, and its data comes from a joystick you hold.
SerpentAI is described as a game agent framework for creating AIs for any game. That is a general toolkit with its own abstractions for game capture and input. TensorKart is the opposite: one game, one capture path, a handful of scripts, and a five-element label vector. If you want to apply the same idea to a different title, TensorKart gives you a reference implementation to copy rather than a framework to configure. Xbox Game AI is noted for using PYXInput to control any Xbox or PC game directly, which again is a broader input strategy than TensorKart's emulator-specific capture.
The honest comparison is scope versus specificity. TensorKart is small enough to read end to end in an afternoon, and that is the point.
Maintenance, licence and what to check before committing
TensorKart is MIT licensed, which permits commercial and private use with the usual requirement to keep the copyright and permission notice. That is a permissive arrangement, and nothing in the supplied material suggests additional restrictions. This is not legal advice; read the LICENSE file in the repository for the operative text.
Maintenance cost is dominated by the environment, not the code. The README's dependency path is apt-get for mupen64plus plus a requirements.txt for Python, with TensorFlow and cuDNN named for GPU acceleration. TensorFlow's Python API has moved substantially since projects of this style were written, and there are no releases retrieved for this repository, so you should expect to resolve version conflicts yourself rather than pull a tagged artifact. The last push is recent, but the absence of releases means there is no versioned upgrade path to follow.
Before adopting it, check three things: that mupen64plus is configured with the sdl input plugin, that your recorded `data.csv` rows correspond to actual emulator frames rather than desktop captures, and that the TensorFlow version in requirements.txt still installs against your Python. Those three checks cover the failure modes the README itself flags.
Who should pick this up
TensorKart fits a specific reader: someone learning end-to-end imitation learning who wants the smallest complete example, from recording through training to emulator playback, in a single repository. The five-element `y` vector and the `utils.py prepare` step make the input-output contract explicit, which is more than many demo projects offer. The LB override in play.py means experiments are recoverable without restarting the emulator.
It does not fit anyone who needs reproducible results across machines, since the dataset is whatever you drove and the capture depends on window placement. It also does not fit anyone hoping for a reinforcement learning baseline out of the box; the README lists that as future work, with `-1` per time-step described as an available reward signal but not a training objective in the current code. The practical next step is to record one short session, run `python utils.py viewer` on it, and confirm the frames look like the game before you invest an hour in train.py.
Editorial conclusion
Adopt TensorKart if you want a small, readable end-to-end imitation learning example that runs against a real emulator and you accept that every training sample comes from your own manual driving. Do not adopt it if you need a headless, reproducible benchmark or a framework that supports multiple games; the play loop depends on gym-mupen64plus, and the README's own notes warn that some captured screenshots can be the desktop instead of the emulator. Verify first that mupen64plus is using the sdl input plugin and that your recorded data.csv rows actually point at emulator frames, because a misaligned capture poisons the whole training set before train.py ever runs.
Community notes