Sycamore: a Ray-backed DocSet pipeline for PDFs with tables, figures and OCR

The problem Sycamore targets: PDFs whose meaning lives in the layout

Most document pipelines fail at the same place. A PDF containing a financial table, a chart with axis labels, or a scanned manual gets flattened into a stream of characters, and the relationship between a number and its column header disappears. The README states that Sycamore can analyze and chunk complex documents such as PDFs and images with embedded tables, figures, graphs, and other infographics. That is the specific gap it addresses: not text extraction in general, but preserving semantic structure so that a later retrieval step returns a chunk that still makes sense on its own.

The audience is narrow but real. If you are building RAG over enterprise reports, presentations, transcripts or manuals, and your evaluation keeps failing on table questions, Sycamore is aimed at you. If your corpus is already Markdown or well-formed HTML, the partitioning stage is overhead with no payoff. The project describes itself as an engine for ETL, RAG, LLM-based applications and analytics on unstructured data, which is broad phrasing, but the features list is concrete about where the effort went: segmentation, table extraction, OCR and visual summarization.

DocSets and the Ray backend: what the abstraction actually buys

The central abstraction is the DocSet, described in the README as a scalable abstraction for document processing that includes high-level transformations in Python for data processing, enrichment and cleaning. The dataflow diagram in the repository shows documents entering, passing through partitioning and transforms, and exiting into a target store. The README frames DocSets as a functional programming approach, which matters for a practical reason: transforms compose, so you can swap a chunking strategy without rewriting the loading logic.

Underneath, the README names Ray as the scalable backend. That choice explains both the appeal and the friction. Ray gives you distributed execution across a cluster without you writing the scheduling code, which is how the project claims to remove what it calls the undifferentiated heavy lifting of reliably loading chunks. The cost is that Ray is a substantial runtime dependency with its own operational model. The README does not document cluster sizing, memory requirements or failure recovery behaviour, so treat the scaling story as a claim about architecture rather than a documented operational recipe.

One thing the README does state plainly is the accuracy claim behind the partitioning model: the Aryn deep learning DETR model, trained on 80k+ enterprise documents, is said to lead to 6x more accurate data chunking and 2x improved recall on hybrid search or RAG compared to alternate systems. Those numbers come from the vendor and no methodology is given in the README. They are a reason to run your own retrieval evaluation, not a reason to skip it.

Partitioning runs through Aryn DocParse, and that is the load-bearing dependency

For processing documents, the README says Sycamore leverages Aryn DocParse, formerly the Aryn Partitioning Service, described as a serverless, GPU-powered API for segmenting and labeling documents, doing OCR, and extracting tables and images. DocParse takes documents in and returns partitioned output as JSON, which Sycamore then feeds through extraction, enrichment, transforms and cleaning before loading downstream.

This is the design decision that shapes everything else. The hardest part of the pipeline, the vision model that finds tables and reading order, is not in the repository. It is an API call. The README does offer an escape hatch: you can sign up for a free API key, or choose to run the Aryn Partitioner locally. The local option removes the service dependency but not the model dependency, and the README does not describe the hardware the local partitioner needs. If you are in an environment where documents cannot leave your network, the local path is the only viable one, and you should confirm its resource profile before designing around it.

The README also notes that you can choose the LLMs used with the transforms. That is a genuine degree of freedom, but it also means quality varies with your model choice, and the README gives no guidance on which models were used for the visual summarization or table extraction results.

Getting it running: pip extras, connectors and an API key

Sycamore currently runs on Linux and Mac OS. Installation is a single command:

pip install sycamore-ai

Database connectors are installed as Python extras rather than bundled. The README gives this example:

pip install sycamore-ai[duckdb]

The supported connector names listed in the README are duckdb, elasticsearch, opensearch, pinecone, qdrant and weaviate. Installing one extra does not install the others, so a pipeline that writes to OpenSearch and reads from DuckDB needs both extras resolved explicitly. That is a reasonable packaging choice, but it means your dependency manifest grows with each target store you add, and the README does not publish a compatibility matrix tying connector versions to store versions.

To use Aryn DocParse you sign up and use the API key, per the README. The README does not name the environment variable or config key that holds that key, so you will need the documentation at sycamore.readthedocs.io for the exact setting. The repository also ships an example notebook, notebooks/sycamore-tutorial-intermediate-etl.ipynb, which the README points to as the intermediate ETL walkthrough. For a first run, the notebook is the more useful starting point than the README, because it shows a working sequence rather than a feature list.

Where Sycamore is the wrong tool

The clearest mismatch is a corpus that is already structured. If your source documents are clean text, HTML or Markdown, the partitioning stage adds a network round trip per document and a vendor relationship without changing the chunks in any meaningful way. Sycamore's value is concentrated in the vision step, and removing that step leaves an orchestration layer you could replace with a few dozen lines of Python.

The second limitation is operational. The README describes the DocParse API as serverless and GPU-powered, which means throughput and latency are outside your control, and the README does not publish rate limits, quotas or a service level. A batch job over a large archive will be paced by an external API. If your pipeline has a hard deadline or a strict per-document cost ceiling, model that before you build on it.

The third is documentation depth. The README covers installation and connectors well and covers operations barely. There is no material in the supplied README on error handling for malformed PDFs, on partial failure during a Ray job, or on how to resume a run that died halfway. Those are the questions that decide whether a pipeline survives contact with a real archive, and they are not answered here. The readthedocs site may cover them; the README does not.

Versus a plain text extractor plus a chunker

The obvious alternative is the conventional stack: a text extraction library to pull characters out of the PDF, then a recursive or token-based splitter, then embeddings and a vector store. That approach is free, has no external service, and is trivial to run locally. The difference in approach is where structure is decided. A text extractor emits a linear character stream and the splitter guesses boundaries from length or punctuation, so a table becomes a run of numbers with no headers attached. Sycamore decides boundaries from the visual layout first, using a segmentation model, and only then chunks. That is why the README can claim better chunking accuracy: the boundary decision happens after layout analysis rather than before it.

If your evaluation shows that table and figure questions are a small share of queries, the conventional stack wins on cost and simplicity. Sycamore earns its keep only when layout-aware chunking measurably changes retrieval results on your own documents. That is a test you can run, and you should run it before adopting the dependency.

Maintenance, licensing and the cost of the API in the middle

Sycamore is Apache-2.0, which permits commercial use and modification, and the repository is not archived. The release history is worth reading carefully. v0.1.32 landed in April 2025, v0.1.33 in July 2025, and v0.1.34 in May 2026. That is a gap of roughly ten months between the last two releases, and the version line is still 0.1.x, which typically signals that interfaces may still change. The repository shows a push in August 2026, so work continues, but the tagged release cadence is slower than the push activity suggests. Plan upgrades around the tags, not the commit stream.

On cost, the material supports one firm statement and one open question. The firm statement: the hosted Aryn DocParse path is a paid service with a free sign-up tier, and the README does not publish pricing, so per-document cost is something you must confirm with the vendor. The open question is the local partitioner, which removes the API bill but moves the GPU cost onto your own infrastructure. Apache-2.0 covers the Sycamore code; it does not cover the DocParse service, and the README does not describe the licence of the DETR model weights hosted on Hugging Face. If you plan to redistribute the model or run it in a commercial product, check that licence separately. This is a factual gap in the supplied material, not legal advice.

Editorial conclusion

Adopt Sycamore if your corpus is layout-heavy (tables, infographics, scanned pages) and you are willing to make Aryn DocParse part of the pipeline, either as the hosted service or as a local run of the Aryn Partitioner. Skip it if your documents are already clean text or simple HTML, because the partitioning stage buys you little and adds a service dependency. Before committing, verify the current pricing and API-key terms for Aryn DocParse, confirm which of the six connectors matches your target store, and check the release cadence yourself: the gap between v0.1.33 and v0.1.34 is roughly ten months, and the project is still on a 0.1.x version line.

Official sources

1. aryn-ai/sycamore on GitHub
2. License: Apache-2.0
3. Project website
4. README
5. Releases