piflow: a DeepAgents research prototype wrapped around a data flow engine

What the project actually is

The README describes it as a research data processing prototype built on DeepAgents, FastAPI, skill integration and a React frontend, and the repository description calls it a big data flow engine with spark support.

Those two descriptions are worth holding side by side, because the project is in a transition between them. The top-level entries show a piflow_engine/ directory, a runtime/ directory with a DAG manager and a piflow bridge, schemas/dag/ with definitions for node, edge, skill, task, binding and obs, and routers for a DAG panel and a DAG runtime. So the flow engine has moved from a separate bundled artefact into the application's own schema and routing layer.

The Overview section lists what the prototype does. Streaming agent chat via /chat/stream. Conversation history management, with create, switch, delete and load for threads. Workspace file handling, uploading inputs and downloading outputs. Message-level attachments, persisted for both uploaded files and preset example files. Example pipelines, where you create a new thread and start a task in one click. And a skills centre that lists skills and categories.

The primary language is Python, the licence is not a standard identifier, the default branch is master, and the recent releases are v2.0.1 for enhanced data connection and workspace integration from 2026-07-31, v2.0.0 for user-isolated workspaces and on-demand operator generation from 2026-07-02, and v1.9.5 from 2024-12-06. The last push was on 2026-10-22 with the project not archived.

The version numbering tells a story of its own. A 1.9 to 2.0 jump in the space of four months, with the release names naming two substantial architectural changes, is what a project looks like when it stops being a script and becomes a platform.

The engine is installed first, and that is the whole setup order

The Prerequisites section is where the project's shape becomes clear, and the first instruction is the one that would confuse anybody arriving from a normal Python package.

Python 3.10 or newer is recommended. Then: install the bundled PiFlow Python engine first, and the command is a direct install of a wheel from inside the repository:

pip install ./third_party/piflow/piflow_engine-0.1.1-py3-none-any.whl

Then install the backend dependencies:

pip install -r requirements.txt

So the flow engine is not on PyPI. It is a py3-none-any wheel, which means a pure-Python wheel that works on any platform and any Python 3, and it ships inside third_party/ along with a lib/ directory described as vendored wheels.

That ordering is not cosmetic. The application imports the engine, so the engine has to be present before the application package resolves, and if you install the requirements first you get a set of packages that import a module that does not exist yet.

The frontend is the third step, and it is separate:

cd vue-web
npm install

Node.js 18 or newer is recommended for the frontend, and the vue-web/ directory has its own package.json, so the frontend is a separate dependency tree with its own lockfile and its own build.

Then PostgreSQL, and the documentation is specific about what the database is for. The project uses PostgreSQL for chat threads, messages, message attachments and skill metadata. Tables are created automatically on startup, but you must prepare the database first and configure config/database.yaml, and the example configuration is:

host: "127.0.0.1"
port: 5432
user: "postgres"
password: "123456"
name: "flow_agent"

That last point, that the tables are created automatically but the database is not, is the kind of detail that saves an hour. The application provisions its schema on start, so the only thing you have to do is have a database and point the config at it.

The LLM configuration comes from config files and environment variables, with the provider API keys in .env. The example is:

DASHSCOPE_API_KEY=your_key
OPENAI_API_KEY=your_key

And there is a third configuration file worth knowing about, config/mineru.yaml, which holds the API key for the MinerU skill, so the document-extraction capability is configured separately from the model providers.

The optional dependency is pandoc, for more complete document conversion support, and pandoc appears in the requirements list as well.

The dependency list, and what it tells you the prototype does with documents

The requirements file is long and it reads as a document-processing stack rather than a chat application.

deepagents~=0.5.1
fastapi~=0.128.0
fastmcp~=3.1.0
langchain-core~=1.2.27
langchain-openai~=1.1.11
pydantic~=2.12.5
PyYAML~=6.0.3
uvicorn~=0.40.0
pytest~=9.0.2
requests~=2.32.5
pycryptodome~=3.23.0
pypdf~=6.10.0
pandoc
python-docx~=1.2.0
defusedxml~=0.7.1
psycopg2-binary~=2.9.11
pdf2image~=1.17.0
webdataset~=1.0.2
transformers~=4.57.5
langchain~=1.2.15
langgraph~=1.1.6
fonttools~=4.61.1
pdfplumber~=0.11.9
pandas~=2.3.3
pillow~=10.4.0
lxml~=6.0.2
python-pptx~=1.0.2
openpyxl~=3.1.5
chardet~=5.2.0
numpy~=1.26.4
datasets~=4.6.1
passlib[bcrypt]~=1.7.4
pyjwt~=2.11.0
bcrypt==4.0.1
minio~=7.2.7
py-data-juicer

The agent and API core is small and current: deepagents at 0.5.1, fastapi at 0.128.0, langchain-core at 1.2.27, langchain-openai at 1.1.11, langchain at 1.2.15, langgraph at 1.1.6, pydantic at 2.12.5, uvicorn at 0.40.0 and fastmcp at 3.1.0. The fastmcp dependency is the MCP server side, which pairs with the mcp_runtime/ directory in the tree.

Then the document handling, and it is thorough. pypdf, pdfplumber, pdf2image, pypandoc-adjacent pandoc, python-docx, python-pptx and openpyxl cover PDF, Word, PowerPoint and Excel. lxml and defusedxml do the XML work, and defusedxml being pinned specifically is a security choice rather than a convenience one. webdataset handles sharded datasets, and datasets at 4.6.1, transformers at 4.57.5 and numpy at 1.26.4 are there for the data side. pandas at 2.3.3, pillow at 10.4.0 and chardet at 5.2.0 cover tabular and binary data and encoding detection.

Then the storage and platform pieces. minio at 7.2.7 is the S3-compatible object store client, py-data-juicer is the data processing library, which is the component that makes the spark support claim plausible, and requests and PyYAML are ordinary. pytest at 9.0.2 is the test runner.

And the security pair at the end: passlib with bcrypt, bcrypt pinned exactly at 4.0.1, and pyjwt at 2.11.0. The exact bcrypt pin is the kind of thing that matters, because bcrypt is a native extension and newer versions have caused build failures, so a project that pins it exactly is a project that has been through that.

Taken as a whole, the dependency list says this is a prototype for taking documents in, running them through an agent that can call tools and a data flow engine, and producing outputs. That is the shape of a research data processing system with an agent interface bolted on, and the storage/ directory with its artifacts, outputs, temp and logs subdirectories is consistent with it.

The repository structure, and the layers it reveals

The structure block in the README is long and it is the clearest description of the architecture available.

The entry points come first: server.py as the FastAPI entrypoint, main_cli.py for the CLI and deep_agent_main.py for the deep agent. The runtime/ directory holds the engine, the chat store, the DAG manager, the skill manager and the piflow bridge, which is the layer where the flow engine and the agent meet.

The agents/ directory contains the agent factory, prompts, middleware and tools, and the tools/ tree is split into core for the tool base, naming and registry, adapters for the DeepAgents and MCP adapters, and a directory whose name is spelled excutor, which looks like a typo for executor, holding executor utilities.

The services/ directory is described as the business logic layer, covering auth, the DAG panel, the DAG runtime and the user, and the routers/ directory holds the matching FastAPI routers. That pairing, service plus router per domain, is a standard and sensible FastAPI arrangement, and it keeps the HTTP layer thin.

The mcp_runtime/ directory is interesting on its own: MCP lifecycle, health, reconnect and a schema cache. An MCP client that caches schemas and can reconnect is doing more than a thin wrapper, and the presence of health and reconnect handling suggests the tool expects the MCP servers it talks to to be flaky.

The schemas/ directory holds auth and user schemas plus a dag/ subdirectory with node, edge, skill, task, binding and obs definitions. Six record types for a directed acyclic graph, which is enough to model a pipeline where nodes are operators, edges are dependencies, skills are the reusable capabilities, tasks are the units of work, bindings are how a node is wired to another, and obs appears to be the observability data.

The workspace/ directory is the runtime root and it contains the state you would expect: skills/ with skill definitions carrying a SKILL.md, dag_system_node/ for the system DAG node operators, and artifacts/, outputs/, temp/ and logs/. The presence of a SKILL.md file per skill is worth noticing, because it is the same convention the Claude skills ecosystem uses, where a skill is a directory with a markdown file describing what the agent should do and how.

The config/ directory has app.yaml, database.yaml, llm.yaml, mcp_servers.yaml, mineru.yaml and default_user.yaml, and the last of those is a template for the workspace isolation introduced in the 2.0 release.

The rest of the tree is the usual supporting cast: infra/ for config, env, logging and settings, database/ for the PostgreSQL connection manager, repositories/ for data access, security/ for JWT, the password handler and the auth dependency, storage/ for static assets, skill icons and conversation history, scripts/ for release scripts, test/ for the Python tests, and third_party/ and lib/ for the vendored wheels.

The frontend is vue-web/ with the usual Vite and Tailwind configuration and a src/ tree of components, lib and pages, so the React reference in the README's summary line is a leftover, and the actual frontend is Vue.

Running it, and the three ways to start the stack

There are three documented ways to run the project, and they differ in what you are testing.

The first is Docker. You need Docker and the Docker Compose plugin, and the version check is:

docker --version
docker compose version

You set the API key before startup, and then start the services:

docker compose -f docker/docker-compose.yml up -d --build

Stopping them is the matching compose command with down, and the logs come from:

docker logs -f piflow-app

The container name in the log command, piflow-app, is a small but useful detail, since it tells you what to look for when the compose file has several services.

The quick check after startup is a single request:

curl http://127.0.0.1:5174/

The port 5174 for the check differs from the 5173 given for the frontend in development, which is a hint that the Docker deployment serves the built frontend from the application server rather than from a separate dev server.

The Docker path also tells you where the image lives, registry.cn-hangzhou.aliyuncs.com/cnic-piflow/piflow:2.0.0, and that docker/config/database.yaml and docker/config/llm.yaml can be edited on the host with changes taking effect after restarting the containers. The config is mounted rather than baked into the image, which is the right way round for anything holding credentials.

The second route is running the backend directly, and the two alternatives are:

python -m uvicorn server:app --host 0.0.0.0 --port 8080 --reload

or:

python server.py

The uvicorn form is the one you use while developing, with --reload picking up code changes, and it binds to 0.0.0.0 on port 8080, which is what you need in a container. The API base is then http://localhost:8080.

The third piece is the frontend, in its own terminal:

cd vue-web
npm run dev

The frontend dev server is then on http://localhost:5173, and if the backend base URL has to be different you create vue-web/.env.

So the development shape is two processes and a database, and the Docker shape is one command. The frontend talking to the backend across two ports on development is the normal arrangement for a Vite application, and the .env file for the base URL is what makes the same built frontend work in the containerised deployment.

How to evaluate it, and what to check first

There are five things to consider, and the first one is about what a prototype means for you.

The README calls it a research data processing prototype. That is a specific and useful word, because it tells you the project is not a supported product with a compatibility promise. The version numbers help, since it is at 2.0.1 with 2.0.0 having landed only a few months before and the release names naming user-isolated workspaces and on-demand operator generation, so the architecture is still moving under you. If you are building on it, pin a version and expect to read the release notes.

Second, the installation order, which is unusual and easy to get wrong. The flow engine is a bundled wheel installed from third_party/ before the application requirements, and the frontend is a separate Node tree. Get that order wrong and the application imports a module that is not there yet. The README gives the correct order, but a scripted install has to follow it literally.

Third, the database and the config files. PostgreSQL is required and holds chat threads, messages, attachments and skill metadata, and the tables are created on startup, so the only manual step is creating the database and filling in config/database.yaml. The model provider keys go in .env, and the MinerU key goes in its own config file.

Fourth, the breadth of the document stack. This is not a chat wrapper around one model. The requirements cover PDF, Word, PowerPoint, Excel, tabular data and sharded datasets, with two separate PDF libraries, defusedxml for XML, and an S3 client for object storage. If you need those formats, that is the project's strength. If you need something else, you are starting from a very large base.

Fifth, the parts of the tree that suggest a specific operational model. The user-isolated workspaces from the 2.0 release, the workspace/ directory with per-skill SKILL.md files, the MCP runtime with a schema cache and reconnect handling, and the reset-style tools are all signs of a system intended to be run per user, with isolation between them, and with external MCP servers that come and go.

What you get is a substantial and coherent prototype with a real backend, a real frontend, a flow engine and a document pipeline, and the honest caveat is the one the project makes itself: it is research code, it moved to 2.0 recently, and the engine ships as a vendored wheel rather than a released package, so you own the upgrade of that component too.

Editorial conclusion

Adopt piflow if you are prototyping a research data processing system that mixes an agent loop with a flow engine and a wide range of document formats, because the DAG schema, the skills centre and the MCP runtime are a lot of working structure to inherit. Do not adopt it expecting a supported product, since the README calls it a prototype and the 2.0 release with user-isolated workspaces landed only in July 2026, so pin a version and expect the architecture to move. Before you commit, follow the install order exactly, since the bundled flow engine wheel has to be installed before the application requirements, prepare the PostgreSQL database yourself even though the tables are created on start, and read the release notes for 2.0.0 and 2.0.1 to understand what the workspace isolation and the on-demand operator generation actually changed, since those are the two features most likely to affect the shape of your deployment.

Frequently asked questions

Official sources

1. cas-bigdatalab/piflow on GitHub
2. Issues
3. README
4. Releases