sqlitebiter: turning any spreadsheet-shaped file into a SQLite database

Twelve input formats and one output file

The tool's scope is stated in a single sentence that is worth expanding, because the format list is the feature.

sqlitebiter is a CLI tool to convert CSV, Excel, HTML, JSON, Jupyter Notebook, LDJSON, LTSV, Markdown, SQLite, SSV, TSV and Google Sheets into a SQLite database file.

The set covers most of what a data file actually turns out to be. Separated values in three flavours, since TSV and SSV are just different separators. Excel, in the sense of the Microsoft Excel format. HTML, which is what you have when a spreadsheet was exported to a web page. JSON, plus the streaming variants LDJSON and NDJSON and the labelled LTSV format. Jupyter notebooks, which is a thoughtful inclusion, since a notebook cell output is often the actual dataset. Markdown, Mediawiki, and SQLite itself, so you can merge an existing database into a new one. And Google Sheets, which the topic list names as google-sheets.

Two of the entries are about handling awkward input rather than format. Multi-byte character support and automatic file encoding detection. Encoding detection matters more than it sounds, because a converter that guesses wrong on a Shift-JIS or GB18030 file produces a database full of mojibake, and the failure is silent.

The output side is deliberately narrow: one SQLite file. That is a good constraint for a tool like this, because the whole value is in having the result be queryable with the sqlite3 binary, with DuckDB, with a notebook, or with whatever your application already reads.

The URL mode is the interesting one, and it exposes a hard problem

Most of these formats are structured, and the conversion is a matter of reading rows. The URL mode is different, because a web page has no agreed structure, and the documentation is refreshingly honest about what comes out.

The example is converting the Wikipedia page for a comparison of firewalls, and the output shows eleven separate tables, one per HTML table on the page:

$ sqlitebiter url "https://en.wikipedia.org/wiki/Comparison_of_firewalls"
[INFO] sqlitebiter url: convert 'Comparison_of_firewalls' to 'Comparison_of_firewalls_Wikipedia_html1' table
[INFO] sqlitebiter url: converted results: source=1, success=11, created-table=11
[INFO] sqlitebiter url: database path: out.sqlite

Then the schema, and the first table is clean:

CREATE TABLE IF NOT EXISTS 'Comparison_of_firewalls_Wikipedia_html1' (Firewall TEXT, License TEXT, [Cost and usage limits] TEXT, OS TEXT);

and the fifth is not:

CREATE TABLE IF NOT EXISTS 'Comparison_of_firewalls_Wikipedia_html5' ([Features:] TEXT, "Configuration: GUI_ text or both modes?" TEXT, "Remote Access: Web (HTTP)_ Telnet_ SSH_ RDP_ Serial COM RS232_ ..." TEXT, [Change rules without requiring restart?] TEXT, [Ability to centrally manage all firewalls together] TEXT);

The documentation says plainly that the attributes within the converted SQLite database may include symbols as above. That is the real limitation of the URL mode stated in its own output. A page written for humans uses column headings that are questions, with newlines in them, and the tool faithfully reproduces that as column names. The data is there and it is queryable, but the schema is whatever the page said.

That is fine for exploration and for a one-off extraction where you then fix the schema by hand. It is not fine for a pipeline that runs on a schedule, where a small edit to a Wikipedia table changes the column names and breaks the queries downstream.

A source tracking table, and why it is there

Every converted table is accompanied by a metadata table, and the schema in the documentation shows what it holds:

CREATE TABLE IF NOT EXISTS '_source_info_' ("source_id" INTEGER NOT NULL, "dir_name" TEXT, "base_name" TEXT NOT NULL, "format_name" TEXT NOT NULL, "dst_table" TEXT NOT NULL, size INTEGER, mtime INTEGER);

So for every output table the tool records the source it came from, the directory and base name, the format it was parsed as, the destination table name, and the size and modification time of the original file. The source_id is the join key back to that record.

That is a genuinely useful piece of design for a converter, and it is worth understanding why it is there. A conversion that merges many files into one database loses the provenance of each row by default. Once the data is in the table, you cannot tell which of forty CSVs a row came from, and the size and mtime let you re-run the conversion for one file when it changes rather than redoing everything.

The base_name and format_name fields also answer the question of why a particular file produced a particular table, which matters when a directory contains a mix of formats and the tool has already guessed the format from the extension or the content.

The documentation also points to a limitations page specifically for JSON conversion, linked from the format list. That is a hint about where the tool's own authors think the sharp edges are, and it is the page to read before you rely on the JSON path with nested objects.

Packaging, entry points, and the pieces behind the formats

The repository layout shows a project with several distribution channels and a good deal of platform awareness.

At the top level there is a docker directory, a snap directory, an enter_win_build_env.bat for the Windows build environment, a cli.py, and a Makefile with targets for building a Debian package, running the tests through tox, formatting shell scripts with shfmt, generating the readme, and cutting a release.

The setup.py is where the optional dependencies are declared, and they map directly onto the input formats. Google Sheets support pulls in gspread, oauth2client and pyOpenSSL. Mediawiki support pulls in pypandoc. There is an optional ujson requirement, and a pyinstaller requirement of 5.13 or newer for building a standalone executable.

The main requirements are not hardcoded in setup.py at all. They are read at build time from a requirements directory, with separate files for the runtime requirements, the test requirements and the documentation requirements, and the lines are stripped and collected into the install_requires list. That is an unusual and slightly awkward pattern, because the effective dependency set is only visible if you open the requirements directory, but it does mean the pinned versions live in one place per environment rather than being duplicated.

The version is read in the same indirect way, with setup.py executing the contents of the module's __version__.py file to pick up the version and the author details, and the long description read from README.rst.

For configuration, the pyproject.toml sets up black with a line length of 100 and a target version list running from py37 to py312, isort with a known third party list, and a mypy section with the strictness flags turned on, including warn_unreachable, warn_unused_configs and warn_unused_ignores. The pytest configuration points at the test directory and enables the markdown reporting plugin with colour set to auto.

Where a format converter is the wrong tool

Three situations rule it out.

The first is when the mapping needs designing. The column names in the output come from the source, and the column types are inferred. If your CSV header is abbreviated, or if two columns mean the same thing, or if one column needs splitting into three, then the design work is a transformation step, and this tool is not a transformation tool. It converts shapes, it does not reshape meaning.

The second is when the input is a database already. SQLite is on the supported list, which is useful for merging several sources into one file. But if what you want is to read and write a live database, then reaching for a query tool or an ORM is the direct route, and a converter that copies rows into a new file adds a synchronisation problem you did not have before.

The third is when the output has to be reproducible. The URL mode in particular will change underneath you. A page gets edited, a table gets a new column heading, and your pipeline now produces a database with a different schema, and since the column names come from the page, nothing in your own code fails until a query returns nothing.

There is also the question of scale. The tool is designed around files and pages, and a URL mode that fetches and parses a web page is going to be bounded by what the page serves and by how the fetch behaves. Nothing in the documentation describes streaming a large endpoint into a database incrementally, and that is a different problem with a different tool.

What the tool does well is the unglamorous middle of the problem, where you have a pile of files of mixed formats and you want them queryable in one place, and you are willing to look at the resulting schema.

A maintained codebase behind a stale set of tags

The maintenance record is the one thing an evaluator has to interpret, because the two signals disagree.

The recent releases are v0.36.3 on 2023-12-02, v0.36.2 on 2023-07-17 and v0.36.1 on 2022-03-27. The last push to the repository was on 2026-04-13, and the repository is not archived. So there have been commits in 2024, 2025 and 2026 that did not come with a new tag, which means the published PyPI releases and the current source tree are not the same thing.

The badges in the README tell you what the project does about quality. There is a CI status badge for Linux, macOS and Windows, so the test suite runs on all three. There is a CodeQL badge for GitHub code scanning, which is a static analysis service for finding vulnerabilities in code, and its presence on a tool that parses untrusted files is a good sign. And there is a supported Python versions badge.

The documentation is built with Sphinx and published on Read the Docs, with a .readthedocs.yaml in the repository and a docs directory, and the project home page is the readthedocs URL rather than a marketing site. There is a constraints.txt, a tox.ini, a pylama.ini, and a Makefile whose check target runs the tox lint environment and then pip check, so a broken dependency set fails the local check as well as CI.

The practical advice follows from that. If you install from PyPI you get v0.36.3 and whatever it contains, which is the version most users have been on for a while. If you need a specific behaviour that was added after that tag, you need the source tree, and then you are running unreleased code.

Editorial conclusion

sqlitebiter fits someone who has data in a format they cannot query and wants it in SQLite, and it fits a scraper who needs a web table turned into a table without writing a parser. It is the wrong tool when your source is already a database, and it is the wrong tool when the mapping between the source fields and the target columns needs to be designed rather than inferred, because the column names come from the source and the type detection is automatic. Before relying on it, read the limitations page the documentation links from the JSON section, because the documentation itself points there for the cases where conversion does not do what you expected, and check the release you are installing, because the published tags stop at v0.36.3 from December 2023 while the repository has seen commits since.

Frequently asked questions

Official sources

1. License: MIT
2. Project website
3. README
4. Releases
5. thombashi/sqlitebiter on GitHub