POMDPs.jl: one interface for writing POMDP solvers in Julia

A vocabulary rather than a solver

POMDPs.jl is a core interface for working with Markov decision processes and partially observable Markov decision processes on discrete and continuous spaces. What it is not is a solver. The package contains only the core interface for communicating about problem definitions, and everything practical lives in packages built on top of it.

The stated goal is a common programming vocabulary for three things: expressing problems as MDPs and POMDPs, writing solver software, and running simulations efficiently. That third item is not an afterthought, since a belief update loop over a particle filter is where most of the compute in a POMDP application actually goes.

The payoff shows up when you write a solver once. Because the problem is described through the interface, a solver only has to know the interface, and every problem defined in the same way becomes usable with it. The README backs that with a table of supported packages, and the project is the hub of a wider JuliaPOMDP organisation with separate repositories for solver packages and for ParticleFilters, which is the particle filter implementation maintained alongside it.

The ecosystem reach is the other argument for the interface. Python can be used to define and solve problems through the quickpomdps package, or through tables directly via pyjulia, so a workflow that lives in Python can still reach the Julia solver stack. POMDPTools also has two-way integration with CommonRLInterface, and therefore with the JuliaReinforcementLearning packages, and the SymbolicMDPs package provides an interface to PDDL models for anyone whose problem already exists in a planner format.

QuickPOMDPs and the Tiger problem as a worked example

The quick start in the README is the classic Tiger POMDP, written out in full with QuickPOMDP, a macro that lets a problem be defined as a single literal with named fields for states, actions, observations, transition, observation and reward.

using POMDPs, QuickPOMDPs, POMDPTools, QMDP

m = QuickPOMDP(
    states = ["left", "right"],
    actions = ["left", "right", "listen"],
    observations = ["left", "right"],
    initialstate = Uniform(["left", "right"]),
    discount = 0.95,
    ...
)

The problem itself is small and the structure is the point. There are two doors, three actions, two observations, and a discount of 0.95. The transition function returns a Deterministic distribution when the action is listen, because the tiger stays behind the same door, and a Uniform distribution otherwise, because opening a door resets the situation. The observation function returns a SparseCat, the sparse categorical distribution, with a probability of 0.85 for the door the tiger is behind, and the reward function pays 10 for opening the wrong door, minus 100 for opening the right one, at a cost of 1 for listening.

The README notes explicitly that this is not restricted to discrete problems with explicitly defined distributions, so the same interface covers continuous state and action spaces. What the literal syntax does buy you is speed of writing, and a problem that reads like a specification rather than a set of type definitions.

Solving and stepping through with the QMDP solver

The rest of the example shows the two calls that matter. A solver object is constructed, and the problem is handed to it:

solver = QMDPSolver()
policy = solve(solver, m)

Nothing about the solver is specific to the Tiger problem. It takes the problem, and it returns a policy, which is the whole point of the interface.

Simulation then runs through stepthrough, an iterator from POMDPTools that yields the requested quantities for each step of the simulation. The example asks for s, b, a, o, r, meaning the state, the belief, the action, the observation and the reward, and caps the run at ten steps:

rsum = 0.0
for (s,b,a,o,r) in stepthrough(m, policy, "s,b,a,o,r", max_steps=10)
    println("s: $s, b: $([s=>pdf(b,s) for s in states(m)]), a: $a, o: $o")
    global rsum += r
end
println("Undiscounted reward was $rsum.")

The belief is the interesting piece. In a POMDP the agent does not observe the state, so the simulation carries a belief distribution and the example prints the probability the belief assigns to each state. The pdf call in the print statement is how you interrogate a belief under the interface, and the state variable printed alongside is the true state, which the agent never sees.

That loop is the pattern for any evaluation: run the policy, collect the quantities you care about, and aggregate them. The stepthrough iterator is what makes large simulations cheap in Julia, because the loop is written as a comprehension over a lazy iterator rather than as a hand-rolled while loop.

POMDPTools as the standard library

POMDPTools is described as a standard library for the POMDPs.jl interface, providing implementations of commonly used components such as policies, belief updaters, distributions and simulators. It is hosted in the same repository, under the lib directory, rather than in a separate one, which is convenient for anyone reading the interface and the reference implementation side by side.

The categories map onto the parts of a POMDP pipeline that everyone writes and nobody wants to write again. Distributions cover the Uniform, Deterministic and SparseCat types the Tiger example uses, and a belief updater is what turns an observation and a prior belief into a new belief. Simulators are the machinery behind stepthrough, and policies cover the various ways of choosing an action from a belief, including the QMDP policy the example produces.

This split is a design choice worth naming, because it is unusual and useful. The interface package stays small, so it can evolve without breaking every solver, and the components that most users need are maintained in the same place. The cost is that the ecosystem is layered: a project that only needs the interface still ends up depending on POMDPTools for the standard components, and a project that wants to avoid the dependency has to supply its own policies, updaters and distributions.

The repository layout is small and readable for a Julia package: src/ for the interface, lib/ for the hosted sub-packages, test/ for the test suite, docs/ for the documentation, a Project.toml, a logo directory, a code of conduct and a contributing guide.

Platform support and the packages built on the interface

The README is specific about platform support, and the distinction it draws is between native and non-native solvers. POMDPs.jl and all packages in the JuliaPOMDP project are fully supported on Linux. macOS and Windows are supported for all native solvers, and most non-native solvers should work but may require additional configuration.

That distinction is not a quibble. A native solver is one written in Julia and using Julia's own package and compilation machinery, so it builds and runs the same way everywhere the project tests. A non-native solver wraps a C or C++ library, and those tend to need a working system toolchain and a matching build of the wrapped library, which is what the configuration caveat refers to. On Linux, where the continuous integration runs, that is not a problem.

The supported packages table lists the tools separately from the solvers, with POMDPTools and ParticleFilters given their own rows and build status badges, and the wider list of solver packages links out to their own repositories. The documentation site is published for both a stable and a latest version, and the examples section of that site includes visualisations, with a separate gallery of POMDP problems to browse.

For getting started, the README points to a Julia Academy course on decision making under uncertainty with POMDPs.jl, which is the shortest route to understanding what a belief update is in practice, and to the GitHub Discussions tab for questions.

Licence, release history and the gap since v1.0.0

The licence is in a LICENSE.md file at the repository root, and the hosting platform does not classify it, which means it is a custom or unrecognised licence rather than one of the standard identifiers. Anyone building on the package should read that file directly rather than assume a permissive open source licence, since the interface package is meant to be embedded in other packages and the terms therefore matter downstream.

The release history is thin for a project with this reach. The three most recent tags are v0.9.6 from 2023-10-22, v1.0.0 from 2024-06-12 and v1.0.0+docs from 2024-06-13. The last push to the repository was on 2026-10-18, so commits have continued well past the 1.0 release, and the documentation is published from the latest branch separately from the stable one.

The practical consequence is that a pinned release of POMDPs.jl is more than two years old, while the default branch has moved on. For a library whose main job is to define an interface that other packages implement, that is a less severe problem than it sounds, since the interface changes less often than an implementation, but it does mean that a project pinning to v1.0.0 and a project tracking the default branch may see different behaviour from the same solver package.

The version numbers also show a habit worth noting. The 1.0.0 release and a 1.0.0+docs tag on consecutive days suggests the documentation build is versioned with the package, so the tag you pick determines which documentation set matches the code.

Editorial conclusion

Adopt POMDPs.jl if you are writing solvers in Julia, or you have a problem definition you want to run against several existing solvers, and keep a self-contained implementation if your problem is a plain MDP with no belief tracking, since the interface buys you nothing there. Before you plan around it, read LICENSE.md, since the hosting platform does not classify the licence, and check the solver list for your platform, because the README says only the native solvers are supported on macOS and Windows while non-native solvers may need extra configuration. If you need a released version rather than the default branch, note that the most recent tag is v1.0.0 from 2024-06-12.

Frequently asked questions

Official sources

1. Issues
2. JuliaPOMDP/POMDPs.jl on GitHub
3. Project website
4. README
5. Releases