kagglehub: reaching Kaggle datasets and models from Python

What it is, and the two environments

The library provides a simple way to interact with Kaggle resources such as datasets, models and notebook outputs from Python. Three kinds of resource, one interface.

And then there is the paragraph at the top of the readme that is the most important thing in the document, because it says the behaviour differs depending on where you are running.

Inside a Kaggle notebook, the resource is automatically attached to the notebook, it appears under the input panel in the notebook editor, and its files are served from a shared Kaggle resources cache rather than from the virtual machine's disk. Outside a Kaggle notebook, the resource files are downloaded to a local cache folder.

That difference is not a quirk; it is the whole design in one paragraph. The shared cache is what makes the notebook case fast and cheap. A notebook that loads a dataset that is already in the shared cache gets the files from there, and a notebook that opens ten of them does not copy ten datasets onto the disk. And because the cache is shared, the same dataset is downloaded once for the whole environment rather than once per notebook.

The consequence for a user is that the same call has different effects in the two places, and the library hides that difference behind one function. The path you get back is a path to files either way, and your code does not have to care whether they were copied or served.

The consequence for somebody writing a production service is more interesting. If your code runs in a notebook, the files are somewhere you did not choose, and the disk usage is not yours to control. If it runs outside, you get a local cache folder, and the readme has a section on changing where that folder is. That matters for a service with a small disk, or one where the cache directory would end up on a slow mount, or one where several processes must not fight over the same cache.

So the library is small, and the design decisions are in the caching and the authentication. The readme's own description of the project, in the package metadata, is access Kaggle resources anywhere, and the anywhere is doing the work.

Five ways to authenticate, and why there are five

The authentication section is short and its first note settles the most common case: the library is authenticated by default when running in a Kaggle notebook, and authenticating is only needed to access public resources requiring user consent or private resources.

That is a well-judged default. Most notebook users downloading a public dataset never think about credentials, because the notebook environment is already authenticated. And the two cases that do need a token are exactly the two where you should be thinking about credentials: a public resource whose owner has required consent, and anything private.

The five options are, in order of intrusiveness.

The first is a function call that prompts for the token, which is the option for a person at a terminal.

The second is an environment variable, which is the option for a script or a service, and the readme's comment is a reminder that the value is copied from the settings interface.

The third is a token file at a path in the user's home directory, which is the option for a machine where you would rather not put the token in the environment of every process.

The fourth is a secret in the Colab environment, with a named secret, and the readme links an article on adding secrets in both the free and the enterprise version of that environment. That is the option for a notebook, where the token should not be in the notebook's own output.

The fifth is the legacy credentials file, generated from the same settings page under a heading for legacy keys, stored at a path in the home directory. The readme is careful to call it legacy, and the existence of the option is for the people whose tooling still expects that file.

There is also a note that if you use the separate command line tool, you have already configured authentication and can skip all of this. Which is a small but important point: the library and the command line tool share credentials, so a developer who has set the tool up does not have a second thing to configure.

What the five options have in common is that they cover the deployment contexts people actually have. A person at a terminal, a service with a secret, a laptop with a file, a hosted notebook with a secret store, and a legacy environment that has a file it already knows about. The count of options is a symptom of the variety of environments rather than of indecision.

Downloading a model, and the handle grammar

The download example is the clearest illustration of the library's model, and the example it uses is a real published model.

import kagglehub

# Download the latest version.
kagglehub.model_download('google/bert/tensorFlow2/answer-equivalence-bem')

The argument is a handle, and the handle is a slash separated path: an owner, a model, a framework, and a variation. That structure is not decoration, because it is the same structure the site uses, so a model page's address is the argument you pass, and a model you publish uses the same grammar in reverse.

The options then cover the practical variations. You can ask for a specific version by extending the handle with the version number. You can ask for a single file within the model by giving a path. You can force a download even when the files are already in the cache, which is what you want when you know the cache is stale. And you can choose the output directory, and combine that with the force flag to overwrite what is there.

Every one of those options exists because somebody needed it. The force flag is the interesting one, because a cache is a performance optimisation and an optimisation that you cannot override is a correctness problem: a model that has been updated on the site, a stale cache, and a script that silently uses the old weights. Being able to say, explicitly, download it again, is the difference between a cache and a bug.

The output directory option with the force flag is the other one worth noting. Together they mean you can pin a model to a specific location on disk, which is what a deployment wants: the code knows where the weights are, and the deployment can decide whether they come from the cache or from a fresh download, and which of the two it prefers.

And the return value is a local path, which is the interface the whole library is built around. The caller gets a directory and then opens files in it, and the library has done the hard part, which is finding the right version, resolving the file, and dealing with the cache.

Uploading, and the ignore pattern rules

The upload example is the mirror image of the download one, and it teaches the handle grammar from the other side.

A handle for uploading has the shape of a user name, then a model, then a framework, then a variation. The readme gives the shape explicitly with placeholders, and points at the model page whose handle you would use. The point is that the person uploading chooses the namespace, and the platform checks it, so the handle is a claim about ownership as much as a location.

The upload takes a local directory and does the obvious thing with it. But the readme then adds three optional arguments, and each is a decision.

Version notes are optional, and they exist because a model's history is a sequence of versions and a version with no note is a version nobody can choose between later. If a model has five versions and two of them have notes, those are the two you can reason about.

A licence name is optional, and that one is more than metadata. A model published without a licence is a model with unclear terms, and the parameter exists so that publishing one is a deliberate act rather than an omission.

And ignore patterns are optional, with a stated default set that they are combined with, and the combination is the important part. The defaults handle the files that every machine learning directory has and nobody wants to publish: the training data, the checkpoints, the temporary files, the version control directory. Your patterns are added to that list rather than replacing it, so a user who ignores one directory does not accidentally publish the training set.

The pattern syntax has one rule that is worth stating, because it is the kind of thing that surprises people: to ignore an entire directory, include a trailing slash in the pattern. Which means that a pattern without a trailing slash matches a file or a directory of that name but the directory rule is different, and the difference exists because ignoring a directory and ignoring the files inside it are not the same operation when the directory contains other directories.

So the upload is a small, well-designed API: a handle, a directory, and three optional arguments that each close a specific hole.

Loading datasets, and the adapter idea

The dataset loading section introduces an adapter, and the adapter is the interesting design decision in this part of the library.

A dataset is a set of files in a directory, and what you want from it depends on the files. A comma separated file wants to be a data frame. A parquet file wants to be a data frame, and a lazy one if you are going to filter it. A folder of images wants to be a dataset object that can be streamed. The library offers three adapters, and the choice is between a data frame library, a lazy frame library, and the dataset library from the model hub.

Each adapter is a named constant, and each maps to a type: the data frame one to a data frame, the lazy one to either a lazy frame or an eager one depending on the files and settings, and the model hub one to a dataset object. The readme is careful to say that the last two can produce multiple objects in certain cases, which is the same honesty about the edge cases that appears throughout the document.

And the optional dependencies are the part that shows the design is right. The data frame adapter needs the data frame library, the lazy one needs the lazy frame library, and the model hub one needs the dataset library. None of them is a hard dependency of the package, so a user who only wants to download a model does not install a data frame library, and a user who wants the data frame adapter installs it with an extra.

The data frame adapter is the one with a table, and the table is a mapping from file extension to the reading method. Comma separated and tab separated files go to the comma separated reader, the JSON ones to the JSON reader, XML to the XML reader, and parquet to the parquet reader. The reason for the table is worth noting: it is not that the library implements the parsing, it is that the library decides which parser to call based on the file extension, which is the decision a user would otherwise have to make by hand for every file in every dataset.

So the adapter is a thin layer that does three things: it looks at the files, it picks a reader, and it returns an object of the type you asked for. The heavy lifting is done by libraries you already know, and the value is in the choice.

The packaging, the test harness and the container

The repository layout is the usual one for a Python package, with a few entries that are worth a sentence.

The build uses a single modern build backend, and the version is dynamic, taken from the package's own initialisation file rather than duplicated in the configuration. The dependencies are small: an HTTP client, a progress bar library, a version comparison library, a YAML parser, and the generated client for the platform's own service, which is pinned to a range with a comment in the configuration telling the maintainer to keep it in step with the command line tool. That comment is a small but important piece of engineering: the library and the tool share an interface, and the pin is what stops them drifting apart.

The optional dependency groups are the four we already met, and the signing one is worth a note. It pulls in a signature library and a transparency log client, which means the library can sign what it uploads and publish the signature somewhere verifiable. For a model host, being able to say that a published model was signed by a known key is a meaningful property, and making it an extra rather than a default keeps the common path light.

There is also a pinning of the package metadata version in the build configuration, with a comment explaining that the publishing tool does not yet support the newest metadata version and that the pin should be removed once it does. That is the kind of comment that saves the next person an hour, and it is also a reminder of how much of packaging is waiting for other people's release cycles.

The test configuration is more revealing. The test environment includes a web framework, an extension for token authentication in that framework, and a JSON web token library with the cryptographic extras. A library that downloads things does not normally need those, so their presence tells you the test suite runs against a locally started server that issues tokens, which is how you test a client without hitting the real service on every run. The environment also includes the libraries for reading spreadsheet files through the data frame readers, and a fast reader for the other data frame library, and the matrix runs on two versions of Python.

And the container is the most interesting file in the repository. It takes a Python version as an argument, installs a pinned build tool, and then adds only four files to the image: the project configuration, the licence, the readme, and the package's own initialisation file. The comment explains why: if any of those files changes, a new image build is necessary, so the image is kept minimal to make that rebuild cheap.

Then the step that pays for all of it. The container pre-creates the build tool's environments at image build time, and the comment says what that buys: it drastically cuts the time to run commands with the wrapper script, because the creation of the environments, including syncing the dependencies, is done once during the build and skipped afterwards. That is a very practical optimisation for a development container, and it is the kind of thing that only shows up when somebody has measured how long the test suite takes to start.

Editorial conclusion

kagglehub fits a Python project that needs a model or a dataset at run time rather than at build time, and fits a Kaggle notebook user who wants the same code to work inside and outside the notebook. It does not fit somebody who wants the full command line tooling, since the readme notes that the separate command line tool already configures authentication. Verify first which resource you are fetching and whether it needs consent, because public resources that require consent will fail without a token even though they are public, and that the cache behaviour is what you expect, since a notebook run and a server run will not leave the files in the same place.

Frequently asked questions

Official sources

1. Issues
2. Kaggle/kagglehub on GitHub
3. License: Apache-2.0
4. README
5. Releases