# NCBI Datasets replaces hand-built Entrez queries with data packages > The datasets and dataformat command line tools gather sequence, annotation and metadata for genes and genomes from across NCBI into zip archives that carry their own schema, with a dehydrated mode for bulk downloads. It is a service backed by an open source client, it is rate limited unless you use an API key, and the repository licence is a custom one rather than a standard open source licence. **ncbi/datasets** — NCBI Datasets is a new resource that lets you easily gather data from across NCBI databases. - Repository: https://github.com/ncbi/datasets - Website: https://www.ncbi.nlm.nih.gov/datasets - Stars: 563 · Forks: 70 - Language: Jupyter Notebook - License: NOASSERTION - Published: 2026-10-22 · Updated: 2026-10-22 · Language: en - Canonical page: https://hysenlabs.com/projects/ncbi-datasets ## Two commands: datasets for the data, dataformat for the metadata NCBI Datasets is described as a resource that lets you gather data from across NCBI databases, and the interface is two command line tools. The datasets command downloads biological sequence data across all domains of life, and the dataformat command converts metadata included as part of the data package from JSON Lines format to other formats. The split matters more than it sounds. The data package carries its metadata in a fixed machine readable form, and dataformat is what lets you reshape that into a table without writing a parser. The documented example extracts selected fields for the human reference genome: ```bash dataformat tsv genome --package human-reference.zip --fields organism-name,assminfo-name,accession,assminfo-submitter Organism name Assembly Name Assembly Accession Assembly Submitter Homo sapiens GRCh38.p14 GCF_000001405.40 Genome Reference Consortium ``` So a single call produces a tab separated table with a header row, taken from fields named in the command. The same tool works on a genome package here; the documentation notes that a schematic of the CLI outlines the available commands. The corresponding download for that same genome is one line: ```bash datasets download genome taxon human --reference --filename human-reference.zip ``` The --reference flag is what selects the reference assembly rather than any assembly of the taxon, and --filename names the output archive. That is the whole mental model: choose what you want, get a zip file, then reshape its metadata. ## Dehydrated downloads, and the three step rehydrate Downloading many genomes is a different problem from downloading one, and the answer is a three step dance around a dehydrated archive. You download a dehydrated zip archive, unzip it, and then rehydrate to access the data. For the human reference genome the sequence is: ```bash datasets download genome accession GCF_000001405.40 --dehydrated --filename human_GRCh38_dataset.zip ``` ```bash unzip human_GRCh38_dataset.zip -d my_human_dataset ``` ```bash datasets rehydrate --directory my_human_dataset/ ``` The reason for the extra steps is that a full genome package is large, and the dehydrated form is a placeholder that names the data rather than carrying it. Rehydration is what pulls the actual files in. If you are pulling hundreds of genomes, this lets you stage the requests and the metadata first, then decide what to actually download. The documentation links a how-to page on downloading large genome data packages, which is where the details of that workflow live. The three step shape is worth remembering because the failure mode is confusing if you skip it: a directory that has been unzipped but not rehydrated contains references to data that is not there. ## Four package types, and the data report schema that describes them NCBI Datasets provides sequence, annotation, metadata and other biological data as zip archives called NCBI Datasets Data Package zip archives. Four types are offered: a Gene Data Package, a Genome Data Package, a specialized Virus Data Package, and a Taxonomy Data Package. The virus package being called out as specialized is the hint that these are not interchangeable. A genome package carries assembly and annotation for a genome, a virus package is shaped around viral genomes, and a taxonomy package is a slice of the taxonomy rather than of a sequence. Alongside the data, each package includes data report files containing metadata about the requested records. The data report schemas describe each type of data report, including the available fields, with descriptions and examples. That link is the part to bookmark. The field names you pass to dataformat come from these schemas, not from the tool, and the schema also tells you which fields exist in the first place. The example above uses names like organism-name, assminfo-name, accession and assminfo-submitter, and the second one, assminfo-submitter, is short for assembly info submitter. Reading the schema is how you find the rest of them, and it is also how you know which report type belongs to which package. ## Rate limits, and the API key that doubles them The requests are rate limited, and the number is specific. NCBI Datasets API and command line tool requests are rate limited, with a default of 5 requests per second. By using your API key you can increase that to 10 requests per second. So the limit doubles rather than disappearing. For a single genome or a handful of genes, 5 requests per second is not a constraint. For a genome-scale sweep, it is the thing that determines how your download loop has to be written, and the documentation links both how to get an API key and how to use it with the command line tools. This is the one operational fact worth checking before you write a script. A bulk download that ignores the limit will not necessarily fail outright, it will just be slower than expected, and the failure mode when it does hit the limit is more likely to be a partially filled directory than a clear error. The API key is free and it is the difference between a script that works and one that stalls. ## An open client for a hosted service, and what the repository actually holds The repository is not the product, and it is worth knowing what it is. The top level holds a client directory, a datasets.openapi.yaml, a training directory, a CONTRIBUTING.md, a LICENSE.md and a README.md. The primary language is listed as Jupyter Notebook, which reflects the notebooks used for the work rather than a library you import. The openapi specification file is the interesting artefact, because it describes the HTTP API that the command line tools are built on. The homepage points at the web interface, and the tools and the web interface are two clients of the same service. The project states that the tools are under active development, and asks for feedback through a GitHub issue or by contacting NCBI directly with questions, comments or feature requests. That is the normal posture for a tool maintained by the people who run the database it queries. The licence is the part to look at. The repository carries a LICENSE.md that is a custom licence the platform does not classify as one of the standard ones. The published scientific citation for the work is given in the documentation, listing the author list and the Sci Data paper on exploring and retrieving sequence and metadata for species across the tree of life with NCBI Datasets, from 2024. If you need to cite the tool in a paper, that reference is provided. ## Install with conda, and watch the version numbers drift apart Installation goes through conda, and the documented command installs the CLI tools datasets and dataformat from the conda-forge channel: ```bash conda install -c conda-forge ncbi-datasets-cli ``` The badges in the documentation point at the same conda-forge package, with one for the version, one for the platforms and one for the download count. Other installation options, including other platforms, are covered by the download and install instructions on the NCBI site. Here is a detail that will confuse someone following the documentation literally. The install text says to install the latest version, described as CLI v16.x, while the repository release tags are v18.39.0 from 2026-10-13, v18.38.0 from 2026-09-28 and v18.37.0 from 2026-09-09. The README line is behind the releases. That is not a small thing, because the data report schemas and the field names you script against change with the version, and the schematic the documentation points to is named for a specific schema version, datasets_schema_complete_v18. Trust the tag and the schema documentation over the version mentioned in the install paragraph. ## Conclusion Adopt NCBI Datasets if you pull reference genomes, gene records or taxonomy data from NCBI in any volume, since the data package format and the dataformat converter replace a pile of Entrez queries and hand-assembled metadata files. Do not adopt it for data that is not in NCBI, and be aware that the rate limit of 5 requests per second, doubled to 10 with an API key, shapes how you write a bulk download loop. Before you script against it, check three things: whether you need an API key for the request rate you intend, whether the data report schema for the record type you want is documented, since the field names come from those schemas and not from the tool, and whether the licence file in the repository suits your use, since it is a custom licence that the platform does not classify as one of the standard ones. ## FAQ ### How do I install the NCBI Datasets command line tools? Run conda install -c conda-forge ncbi-datasets-cli, which installs both the datasets and dataformat tools. The README mentions CLI v16.x for this step while the repository release tags are on v18, so check the schema documentation for the version you actually get. ### How do I download the human reference genome with NCBI Datasets? Run datasets download genome taxon human --reference --filename human-reference.zip, which fetches the GRCh38 reference assembly as a data package zip archive. ### How do I download many genomes at once with NCBI Datasets? Use a three step process: download a dehydrated zip archive, unzip it, then rehydrate the directory. The commands are datasets download genome accession GCF_000001405.40 --dehydrated --filename human_GRCh38_dataset.zip, unzip, and datasets rehydrate --directory my_human_dataset/. ### What are the rate limits for NCBI Datasets requests? By default the API and command line tool requests are limited to 5 requests per second. Using your API key raises the limit to 10 requests per second, and the documentation links instructions for getting a key and for passing it to the command line tools. ### What licence does the NCBI Datasets repository use? The repository carries a LICENSE.md that is a custom licence which the platform cannot classify as one of the standard ones, so read that file directly if the terms matter for your use. The project is not archived and the last push was on 2026-10-13. ## Sources - [Issues](https://github.com/ncbi/datasets/issues) - [ncbi/datasets on GitHub](https://github.com/ncbi/datasets) - [Project website](https://www.ncbi.nlm.nih.gov/datasets) - [README](https://github.com/ncbi/datasets/blob/master/README.md) - [Releases](https://github.com/ncbi/datasets/releases) --- Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/ncbi-datasets