Prefect: a Python workflow orchestrator that turns scripts into schedulable pipelines

The problem Prefect solves for Python data teams

Most data pipelines start life as a script someone runs by hand. The script works, then it grows a loop, a retry, a dependency on yesterday's output, and a cron entry that nobody remembers editing. Prefect targets that moment. Its stated goal is to be, in the README's words, "the simplest way to elevate a script into a production workflow." You keep writing Python. The framework supplies scheduling, retries, caching, event-based automations and run tracking around the functions you already have.

The intended audience is data and platform engineers working in Python. The repository layout reflects that: examples include a dbt runner, an API-sourced ETL script, a web scraper, and several AI-agent examples that call out to model providers. Prefect does not ask you to express your pipeline in a separate DSL, and it does not require a scheduler daemon on every machine. A single process can serve a deployment, and the README's own quickstart does exactly that on a laptop.

Where it fits less well is anywhere the pipeline is not Python. The package declares Python 3.10 through 3.14 in pyproject.toml and nothing else. If your transformation logic lives in SQL, a JVM service, or a shell script you cannot wrap, Prefect can still call out to it, but you lose most of the reason to use it.

Flows, tasks and the API-backed state machine

The mechanism is a decorator pair. A function marked with @flow becomes a unit Prefect will track, schedule and retry as a whole. A function marked with @task becomes a unit inside it that gets its own run record, its own retries and its own cache key. Calling a task from inside a flow does not execute it inline the way a normal function call would; the framework intercepts the call, records the intent, and executes it under the orchestration engine.

That engine is backed by a server. Prefect ships a self-hosted server and also offers Prefect Cloud, and the same client talks to either. Run state, logs and metadata are persisted through SQLAlchemy against SQLite or Postgres, which is why aiosqlite, asyncpg and alembic appear in the dependency list. The API itself is FastAPI. When you start a server locally, the UI is served at http://localhost:4200, and the README notes you can run a flow manually from the UI or the CLI, not only on its schedule.

Two design consequences follow. First, the dependency graph is determined at runtime rather than declared up front, so a loop that calls a task ten times produces ten task runs, and conditional branches produce only the branch actually taken. Second, every task call is a round trip to the API unless you disable that behaviour, which matters when you wrap a tight inner loop in @task and wonder why it slowed down. The framework gives you the choice; it does not make it for you.

Installing Prefect and running a first flow

Prefect requires Python 3.10 or newer. The README gives two install commands, one for pip and one for uv:

pip install -U prefect

uv add prefect

The README's example flow fetches a GitHub star count, which is a convenient way to prove the wiring works end to end. Save it to a file and run it with plain Python:

from prefect import flow, task
import httpx

@task(log_prints=True)
def get_stars(repo: str):
    url = f"https://api.github.com/repos/{repo}"
    count = httpx.get(url).json()["stargazers_count"]
    print(f"{repo} has {count} stars!")

@flow(name="GitHub Stars")
def github_stars(repos: list[str]):
    for repo in repos:
        get_stars(repo)

if __name__ == "__main__":
    github_stars(["PrefectHQ/prefect"])

To see the run in a UI rather than only in your terminal, start the self-hosted server. The README says the interface is then reachable at http://localhost:4200:

prefect server start

Scheduling is the last step. Replacing the __main__ block with a serve call turns the script into a long-running process that watches for scheduled deployments. The README's example schedules the flow every minute with a cron expression:

if __name__ == "__main__":
    github_stars.serve(
        name="first-deployment",
        cron="* * * * *",
        parameters={"repos": ["PrefectHQ/prefect"]}
    )

What you should see is a process that stays in the foreground, registers a deployment named first-deployment, and fires the flow once a minute. The README notes you can also trigger that deployment manually from the UI or the CLI, and that deployments can be started in response to events. If you only want the client side, for example inside a short-lived container that talks to an existing server, the README points to a separate prefect-client package described as a lighter-weight option for ephemeral execution environments.

Where Prefect gets in the way

The runtime graph is the trade-off. Because dependencies are discovered as the flow executes, you cannot validate the whole pipeline before it starts. A typo in a branch that only runs on the last day of the month will not surface until that branch runs. Teams coming from a declarative scheduler often expect a dry run that proves the DAG, and Prefect does not offer that in the way they expect.

Operationally, the server is a stateful component. The README documents starting a server but does not document rollback, and the repository carries alembic migrations, which means schema changes ship with releases. Anyone running the self-hosted server should treat the database as something to back up before an upgrade rather than something to recreate. The project also publishes nightly development releases such as 3.8.5.dev2 alongside stable ones like 3.8.4, so an unpinned install can land on a development build.

There is also a scale boundary. The README says Prefect Cloud automates over 200 million data tasks monthly, which describes the hosted product, not a self-hosted server running on a laptop. A self-hosted instance uses SQLite by default, and SQLite is the wrong database for many concurrent workers writing run state. If you are running hundreds of flows per hour, plan for Postgres from the start rather than migrating later. Finally, if your team does not write Python, the framework's main advantage disappears and you are paying for an orchestration server without using the part that makes it pleasant.

Prefect compared with Airflow

The comparison people search for is Prefect versus Airflow, and the difference is where the graph is defined. Airflow asks you to declare a DAG as an object with explicit task dependencies, and the scheduler reads that declaration to decide what to run. Prefect asks you to write a function and call other functions; the graph is whatever your code did.

That changes three things in practice. Testing is one: a Prefect flow is a Python function you can call directly in a test, while an Airflow DAG is typically exercised through the scheduler or a task runner. Dynamic fan-out is another: a Prefect flow can decide at runtime how many tasks to create based on a query result, which in Airflow usually means a more elaborate construct. The third is the scheduler itself. Airflow's scheduler is a central component you run and monitor; Prefect's deployments can be served by the same process that defines them, which is why the README's quickstart fits in a terminal window.

The cost is predictability. Airflow can show you the entire DAG before anything runs. Prefect cannot, because the graph does not exist yet. Neither approach is universally better; they fail in opposite directions. If your pipelines are stable and you want a reviewable structure, Airflow's explicitness is an asset. If your pipelines branch on data and you want to keep writing Python, Prefect's runtime graph is the point.

Licence, maintenance and upgrade cost

Prefect is licensed Apache-2.0, declared in both the LICENSE file and the pyproject.toml license field, with Prefect Technologies, Inc. as the author. That is a permissive licence, and it does not impose copyleft obligations on your own code. It also does not grant trademark rights, and the README distinguishes the open source framework from Prefect Cloud, a commercial hosted product. Teams that need a support contract or an SLA are buying the hosted offering, not the licence. This is a description of the licence text, not legal advice; check with your own counsel if the distinction matters to your organisation.

The last push to main was on 2026-08-28, and the repository is not archived, so development is current. Version 3.8.4 was released on 2026-08-25, followed by nightly development releases on 2026-08-27 and 2026-08-28. The upgrade cost is mostly database migrations on the self-hosted server, since alembic is a direct dependency and schema changes are part of the release process. The README does not document a rollback procedure, so the practical safeguard is a database backup taken before you upgrade, plus a pinned version in your requirements file. The repository also carries a Dockerfile and a Dockerfile.sqlite-builder, which suggests container images are a supported deployment path rather than an afterthought.

Editorial conclusion

Adopt Prefect if your pipelines are already Python functions and you want scheduling, retries, caching and run history without rewriting them as a declarative DAG. Do not adopt it if you need a visual builder, a non-Python runtime, or a fixed graph that can be validated before execution; Prefect decides the graph at runtime, which is the source of both its flexibility and its hardest failure modes. Before committing, verify three things on your own machine: that a flow run survives a worker restart, that your Prefect server database is backed up, and that the version you pin is one you can upgrade from, since the project ships nightly development releases alongside stable ones.

Frequently asked questions

Official sources

1. Official documentation
2. Official README
3. Project repository
4. Release notes