Daft: a Rust data engine that treats images and audio as first-class column types
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
At a glance
- What is it?
- Daft is an Apache-2.0 dataframe engine from Eventual-Inc that runs LLM calls, embedding generation and media decoding inside the same query plan as ordinary tabular transforms. It is a good fit when your pipeline is mostly unstructured bytes with some metadata attached, and the wrong tool when you only need SQL over Parquet.
- Who is it for?
- Adopt Daft if your pipeline decodes images, audio or video and then feeds them to a model, and you want that work expressed in one lazy dataframe plan rather than a hand-rolled Ray or Spark job with a Python UDF in the middle. Do not adopt it if your workload is plain SQL analytics over Parquet or Iceberg with no media or model inference step, because Polars or PySpark will cover that with a smaller dependency surface and a wider hiring pool.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository received new commits within the last day.
- What is it written in?
- Mainly Rust, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem Daft targets: media bytes trapped outside the dataframe
Most dataframe engines assume a row is numbers and strings. The moment a column holds a JPEG, a WAV file or a 1024-dimension embedding, the usual pattern is to drop into a Python UDF, decode inside it, and lose the query optimizer at exactly the point where the expensive work starts. Daft's README frames the project as a high-performance data engine for AI and multimodal workloads, with native multimodal processing listed as the first feature: images, audio, video and embeddings handled alongside structured data in a single framework. That is the specific gap. The audience is data and ML engineers who already write dataframe code but whose rows carry pointers to S3 objects, and who currently glue together a loader, a decoder, a model call and a writer as separate stages. The README's own quickstart description is telling: load a real-world e-commerce dataset, process product images, and run AI inference at scale. The unit of work is a product row, not a tensor batch and not a table partition. If your pipeline already looks like that, Daft is aimed at you. If it does not, the multimodal column types are dead weight.
Python surface, Rust core, Arrow in between
The README describes the split directly: Python-native, Rust-powered, with the pitch that you skip the JVM complexity while Rust sits under the hood. The related-projects table lists Daft as Arrow backed with a vectorized execution engine and out-of-core support, and as the only row in that table marked Yes for query optimizer, multimodal, distributed, Arrow backed, vectorized execution and out-of-core simultaneously. Ray Data is marked Yes for multimodal and distributed but No for a query optimizer; Polars is marked Yes for the optimizer and Arrow backing but No for distributed and multimodal beyond Python objects. That combination is the architectural claim: a logical plan gets optimized before execution, and the physical operators are Rust and Arrow rather than Python objects moving through a scheduler. Out-of-core is the part that matters for media work, because a folder of video will not fit in memory and the engine has to spill or stream rather than assume it will. The README does not document the operator set, the spill policy or the memory manager's thresholds, so treat the out-of-core claim as a capability statement rather than a tuning guide. What is verifiable from the repository layout is the language split: Rust is the primary language and Python is the interface, which means the performance-sensitive path is not the one you write.
Getting it running: pip, Python 3.10, and the Ray extra
The installation instruction in the README is a single command, pip install daft, with an explicit floor of Python 3.10 or higher. The README points to a separate Installation Guide for anything beyond that, specifically installing from source or installing with extra dependencies such as Ray and AWS utilities. So the base wheel and the distributed path are different installs, and the Ray extra is the one that turns a local dataframe into a cluster job. The README links a distributed Ray page and a distributed Kubernetes page under docs.daft.ai, but it does not print the cluster-initialisation calls or the config keys in the README itself, so the exact connection arguments have to come from those pages. One config key is stated outright: setting DO_NOT_TRACK=true disables telemetry. The README says telemetry is collected through Scarf, describes it as non-identifiable and metadata-only, and states that no proprietary code or data is collected and that user data is not bought or sold. The tracking pixel is an image URL pointing at static.scarf.sh, which means an air-gapped or egress-restricted environment will need that variable set or the request will simply fail. That is the full set of concrete setup facts the README provides. Everything else, including how to point Daft at S3, GCS, Iceberg, Delta Lake, Hugging Face or Unity Catalog, is named as a capability but not shown as code.
AI operations inside the plan, not beside it
The second README bullet is the one that separates Daft from a general dataframe engine: run LLM prompts, generate embeddings, and classify data at scale using OpenAI, Transformers, or custom models. The design implication is that a model call is an expression in the plan, so the optimizer can reason about it the same way it reasons about a filter or a projection. That is a different shape from the common alternative, where inference runs in a separate batch job and the results are joined back by key. The README does not specify batching behaviour, retry semantics, rate-limit handling or caching for these operations, and those are exactly the details that decide whether an in-plan LLM call is cheaper or more expensive than a separate inference service. Treat the bullet as a statement of scope, not of production hardening. The same caution applies to the custom-model path: the README names it as an option without describing the interface. If your team already runs a model server with its own queueing and observability, embedding those calls into a dataframe plan may duplicate machinery you have already built.
Where Daft is the wrong choice
The clearest failure mode is a workload with no unstructured data in it. If you are doing aggregations, joins and window functions over Parquet or Iceberg and nothing else, Daft's distinguishing features are unused, and you are taking on a younger engine than the one you would otherwise pick. The README's comparison table is honest about the trade: PySpark is marked No for multimodal, Dask DF is marked No for a query optimizer, Pandas is marked No for distributed and out-of-core. Those engines are not weaker at tabular work; they are weaker at media work. Choosing Daft for pure tabular analytics inverts the reason the project exists. A second limitation is that the README is thin on operational detail. It advertises intelligent memory management and sensible defaults that eliminate configuration headaches, which is a claim about defaults, not a description of what happens when a video decode step exhausts memory on a worker. There is no documented failure-recovery story in the material provided, no statement about exactly-once semantics, and no list of what happens when a model endpoint returns errors mid-plan. A third constraint is the interface itself: the API is Python, and the README gives no indication of a non-Python client. Teams whose pipelines are JVM-centric or Go-centric would be introducing a Python runtime into the critical path, which is the opposite of the JVM complexity argument the README makes. Finally, the version cadence visible in the release list is rapid, with three patch releases inside roughly six weeks. Frequent patch releases are normal for a young engine but they do mean upgrade testing is a recurring cost rather than a one-off.
Polars and Ray Data as the two real alternatives
The README's own table points at the alternatives, and two of them matter. Polars is marked Yes for a query optimizer, Arrow backed, vectorized execution and out-of-core, and No for distributed and for multimodal beyond Python objects. The difference in approach is scope: Polars optimizes single-machine execution hard and leaves distribution to you, while Daft's claim is that the same plan scales from local to a Ray or Kubernetes cluster. If your data fits on one machine after filtering, Polars is the smaller commitment and its optimizer is the part doing the work. Ray Data is the opposite trade. It is marked Yes for multimodal, distributed, Arrow backed and out-of-core, and No for a query optimizer. That is the crux: Ray Data gives you a distributed execution substrate and lets you express media processing as tasks, but it does not rewrite your pipeline before running it, so the burden of efficient execution sits with you. Daft's bet is that the optimizer plus Arrow plus Rust beats hand-scheduled tasks. That bet is plausible for pipelines with filters, projections and joins around the expensive media step, and less relevant for pipelines that are one uniform map over every file. Pick Ray Data when the work is embarrassingly parallel and you want control; pick Daft when the plan has enough structure that an optimizer has something to optimize.
Licence, telemetry and what maintenance actually costs
Daft is Apache-2.0, stated in the README and in the repository metadata. That is a permissive licence with a patent grant, and it means embedding the engine in a commercial pipeline does not by itself create a source-disclosure obligation. This is not legal advice; if you redistribute a modified Daft or bundle it into a product, read the LICENSE file and the NOTICE handling yourself. Telemetry is the other governance detail, and it is opt-out rather than opt-in: data is collected via Scarf unless DO_NOT_TRACK=true is set. For most teams this is a one-line environment variable in a container spec. For regulated environments, the fact that the default is on is worth knowing before the first deploy, not after. On maintenance, the visible signals are the release cadence and the contributor documentation. The README points to CONTRIBUTING.md as the description of the development lifecycle and toolchain, including how to add functionality to the core engine and expose it through the Python API. That matters for cost estimation: extending the engine means writing Rust, not Python, so a team that wants to add a new operator needs Rust skills on staff or a willingness to upstream the change. The README also lists good first issues as a labelled set of entry points, which suggests the project expects outside contributors rather than treating the core as closed. None of this speaks to long-term support windows or a deprecation policy, because the README does not describe one. If you need a versioned LTS commitment, you will not find it in this material.
Editorial conclusion
Adopt Daft if your pipeline decodes images, audio or video and then feeds them to a model, and you want that work expressed in one lazy dataframe plan rather than a hand-rolled Ray or Spark job with a Python UDF in the middle. Do not adopt it if your workload is plain SQL analytics over Parquet or Iceberg with no media or model inference step, because Polars or PySpark will cover that with a smaller dependency surface and a wider hiring pool. Before committing, verify three things against your own data: that the built-in AI operations cover the model providers you actually call, that your Python version is 3.10 or higher, and whether the Scarf telemetry endpoint is reachable from your network or needs DO_NOT_TRACK=true. The engine is Apache-2.0, so the licence itself is not the constraint; the constraint is whether your team wants a dataframe API that is younger than the ones it replaces.
Community notes