Lance: an open lakehouse format for multimodal AI data on object storage

What Lance solves that Parquet and Iceberg do not

Parquet is built around scanning column chunks. Fetching a single row means seeking into many column chunks scattered across files, and that cost is what makes point lookups on Parquet painful. Lance is a columnar file format plus a table format plus a catalog spec, and its stated goal is to keep scan performance while making random access roughly 100x faster than Parquet or Iceberg, according to the README. The same README frames the audience in three groups: people building search engines and feature stores with hybrid search, large-scale ML training that needs random access IO, and anyone storing images, videos, audio, text and embeddings together.

The second problem is metadata. Adding a column to a Parquet dataset traditionally means rewriting files. Lance describes data evolution as adding columns with backfilled values without full table rewrites, which is the feature-engineering case where you keep appending embeddings or derived features to an existing corpus. Versioning is the third: the README says versioning comes with ACID transactions, time travel, tags and branches, with no extra infrastructure. That last clause is the interesting one, because it means the version history lives in the format rather than in a separate catalog service you have to run.

How the format, table and catalog layers fit together

The repository is a Rust workspace with a deliberately layered crate list. Cargo.toml lists lance-file, lance-encoding, lance-io, lance-table, lance-index, lance-index-core, lance-linalg, lance-tokenizer, lance-namespace and lance-namespace-impls as separate members. That split mirrors the pitch: lance-file and lance-encoding handle the columnar layout and blob encoding, lance-table handles the table-level metadata and transactions, lance-index covers vector and secondary indices, and the namespace crates cover the catalog spec.

The Python package is not a workspace member. Cargo.toml excludes python and java/lance-jni and notes that the Python package is built by maturin, which is why pip install pylance pulls a compiled extension rather than pure Python. The Java side is JNI bindings. Indexing is a separate step from writing: the README's SIFT example writes a dataset first, then calls create_index on the vector column, then queries with nearest. That means a freshly written Lance dataset has no vector index until you build one, and search falls back to whatever the unindexed path does until then.

Storage versioning is the part worth reading twice. Each dataset carries a data_storage_version, and the README states that once a dataset is written with a stable version, future Lance releases will keep reading it. SDK and API compatibility is explicitly separate and follows semantic versioning. The next alias is called unstable and is meant for experimentation, never for production data.

Installing pylance and running a first vector search

The documented install is a single pip command. The README gives no separate system dependency step for the Python path, though the Rust side requires rust-version 1.91.0 per Cargo.toml if you build from source.

pip install pylance

The README also documents a preview channel, which it says is published more often than full releases, receives the same level of testing, and is guaranteed to stay downloadable for at least six months. It recommends a stable release when you want to pin a specific version.

pip install --pre --extra-index-url https://pypi.fury.io/lance-format pylance

Converting an existing Parquet dataset is the two-line claim in the project description. The README builds a small Parquet file with pyarrow, opens it as a pyarrow dataset, and hands it to lance.write_dataset.

import lance
import pandas as pd
import pyarrow as pa
import pyarrow.dataset

df = pd.DataFrame({"a": [5], "b": [10]})
uri = "/tmp/test.parquet"
tbl = pa.Table.from_pandas(df)
pa.dataset.write_dataset(tbl, uri, format='parquet')

parquet = pa.dataset.dataset(uri, format='parquet')
lance.write_dataset(parquet, "/tmp/test.lance")

Reading it back returns a pyarrow dataset, which is the integration point for everything else. The README's assertion is that lance.dataset returns an instance of pa.dataset.Dataset, so Pandas, DuckDB and Polars code that already speaks Arrow keeps working.

dataset = lance.dataset("/tmp/test.lance")
df = dataset.to_table().to_pandas()

For vector search the README uses the SIFT1M subset, converts fvecs into a Lance dataset with vec_to_table, then builds an IVF_PQ index on the vector column with num_partitions=256 and num_sub_vectors=16. Queries go through dataset.to_table(nearest={"column": "vector", "k": 10, "q": q}). The README notes twice that DuckDB must be v0.7 or newer or the query can segfault, which is a real constraint rather than a footnote.

Where Lance is the wrong tool

The release history is the first limitation. The three most recent releases are v12.0.0-beta.4, v12.0.0-beta.3 and v12.0.0-beta.2, all dated 2026-08-26 or 2026-08-27, and the workspace version in Cargo.toml is 12.1.0-beta.0. The README's own file format stability section explains why: the project releases frequently because SDKs, integrations and performance work move quickly. That is a deliberate policy, but it means anyone adopting Lance takes on a moving SDK surface. The mitigation the README offers is that the file format is a separate contract from the SDK, identified by data_storage_version.

Mixed-version deployments are the second failure mode, and the README names it directly. Older Lance releases may not understand file format versions introduced later, and the guidance is to pin data_storage_version for deterministic writes when running mixed versions. If your writers and readers upgrade on different schedules and nobody pins that value, you can end up with datasets that some of your fleet cannot read. The next alias makes this worse if someone uses it outside experimentation.

The third case is scan-only analytics on data that already lives in Iceberg. Lance's argument is random access, vector search, full-text search and column backfill. If none of those are in your workload, the format swap buys you nothing and costs you a migration. The README does not document rollback from Lance back to Parquet, so treat conversion as one-way in practice.

Lance against Iceberg and Parquet, and where the difference actually shows

The honest comparison is not Lance versus Parquet as file layouts. It is Lance versus a Parquet or Iceberg table plus the extra systems you bolt on to get the same behaviour. To match Lance's vector search on Iceberg you add a separate vector store and keep it in sync with the table. To match full-text search you add a search engine. To get time travel on plain Parquet you add a catalog with snapshot management. Lance's claim is that the file format, table format and catalog spec are one stack, so the index, the version history and the data sit in the same object storage layout.

The cost of that consolidation is ecosystem gravity. Iceberg has broad engine support and years of operational tooling around it. Lance counters with integrations, and the README lists Apache Arrow, Pandas, Polars, DuckDB, Apache Spark, Ray, Trino, Apache Flink, and open catalogs including Apache Polaris, Unity Catalog and Apache Gravitino. That is a wide list for a format this young, but the README itself hedges with "more integrations on the way", which is a signal that coverage is still filling in rather than complete.

A narrower alternative for the vector half alone is a dedicated vector database. It will typically be simpler to operate for pure similarity search, but it does not give you SQL analytics on the same rows, and it does not give you column backfill without a rewrite. The reason to pick Lance over that is the hybrid case the README leads with: vector similarity, BM25 and SQL analytics on one dataset.

Maintenance, upgrade cost and the Apache-2.0 terms

The last push to the repository was on 2026-08-27, and the most recent releases are betas from the same week, so the project is moving. The README describes it as in active development and points contributors at the contributing guide. For an adopter, the practical consequence is that you should expect to track releases rather than install once. The README's migration guide is where SDK and API changes are documented, and it states that those changes follow semantic versioning, which at least gives you a predictable breaking-change boundary for the Python and Java surfaces.

Upgrade cost splits into two tracks. The SDK track is the one that bites during normal development, because pylance and the Rust crates move together and integrations track them. The storage track is the one that bites in production, and the README's answer is to write with a stable data_storage_version so future releases keep reading your data. Those two tracks being separate is the single most useful thing in the README for anyone planning a rollout.

On licensing, the workspace declares Apache-2.0 and the repository ships RUST_THIRD_PARTY_LICENSES.html plus a Makefile target that regenerates it with cargo about, and a Python equivalent generated through pip-licenses into PYTHON_THIRD_PARTY_LICENSES.md. The Java bindings get their own third-party report through the Maven license plugin. Apache-2.0 is permissive and includes a patent grant, but the third-party reports exist because the dependency tree is large; if your organisation has a licence review process, those generated files are the artefact to hand over rather than the top-level LICENSE alone. This is a description of what the repository publishes, not legal advice.

Testing Lance locally against object storage

The repository ships a docker-compose.yml that stands up LocalStack with S3, DynamoDB and KMS enabled on port 4566, using localstack/localstack:4.0.

services:
  localstack:
    image: localstack/localstack:4.0
    ports:
      - 4566:4566
    environment:
      - SERVICES=s3,dynamodb,kms

That is useful because Lance's premise is a lakehouse built on object storage, and testing that premise against a real S3 endpoint usually means cloud credentials and a bucket. The compose file's own comment notes that LocalStack does not validate the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY values it sets, so they are placeholders. Note also that the file declares version: "3.9", which Compose has deprecated; recent Docker Compose releases warn about it. If you want to check how Lance behaves over object storage before committing to a cloud bucket, this is the cheapest path the repository offers, and the healthcheck on http://localhost:4566/_localstack/health tells you when it is ready.

Editorial conclusion

Adopt Lance when your workload is random access over multimodal rows, vector or BM25 search on the same table, or feature backfills that would otherwise rewrite a whole Parquet dataset. Do not adopt it if you need a frozen API surface, if your data already lives in Iceberg and scan-only analytics is the whole job, or if you cannot pin data_storage_version in every writer. Before writing production data, check the format versioning guide for the current compatibility matrix, confirm the version your pylance install resolves to, and verify that every writer in the pipeline pins the same stable data_storage_version rather than the next alias.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes