Microsoft Psi: a streaming platform for multimodal AI systems

A framework for systems that fuse several live streams

Psi is described in the README as an open, extensible framework for development and research of multimodal, integrative-AI systems, with the short name pronounced like the Greek letter. The examples it gives are multimodal interactive systems, social robots and embodied conversational agents, mixed-reality systems, and applications for ambient intelligence or smart spaces.

The useful summary in the README is the sentence that follows: in essence, any application that processes streaming sensor data such as audio, video or depth, combines multiple AI technologies, and operates under latency constraints can benefit from the affordances the framework provides. That is a narrower claim than a general machine learning framework and a much broader one than a library, and it is the right scope to judge the project by.

The three things the framework provides are named directly: a modern, performant infrastructure for working with multimodal, temporally streaming data, a set of tools for multimodal data visualisation, annotation and processing, and an ecosystem of components for various sensors, processing technologies and effectors. The primary language is C#, the core infrastructure is built on .NET Standard, and the default branch is master with the last push on 2026-10-19. The framework home is the project wiki, and there is a technical report describing it in depth, plus a blog post giving a high level overview.

Streams as the unit of composition

The core concept the tutorial is built around is the stream. The Brief Introduction tutorial, which the README recommends as the starting point, shows how to create a simple program, describes the core concept of a stream, and explains how to transform, synchronise, visualise, persist and replay streams from disk. That list is the architecture in miniature: a Psi application is a graph of components connected by streams, and the interesting capabilities are the operations on those streams rather than the components themselves.

The composition model is what makes the framework more than a set of sensor drivers. A component that consumes audio and a component that consumes depth can be joined, and the join is a first-class object with its own semantics, which is how a system that must react to a voice and a gesture at the same time is expressed. The visualisation, annotation and processing tools then operate on the same graph, so the recorded data and the live data go through the same visualisation path, and a stream persisted to disk can be replayed later as if it were live.

The repository topics reflect that emphasis, with stream-processing and streaming alongside multimodal, perception, human-robot-interaction, pipelines, component-library, framework and artificial-intelligence. The AI in Psi is a component type rather than the point of the project, and a reader who expects a model zoo will not find one here.

Getting started: NuGet packages or build the source

The README splits the getting started path in two, and the choice matters for how you consume the framework.

The core Psi infrastructure is built on .NET Standard and therefore runs both on Windows and Linux, but some components and tools are more specific and are available only on one or the other operating system. You can build Psi applications either by using the Psi NuGet packages, or by cloning and building the source code, and the README links a wiki page for each route.

The source tree in the repository is arranged as a Visual Studio solution, Psi.sln, with a Sources/ directory for the code, an Applications/ directory, a Build/ directory with the build scripts, and build.sh at the root, alongside Directory.Build.props for shared build properties and a NuGet.Config for package restore. There is a QUICKSTART.md at the top level, and a separate .msr/ directory for the Microsoft Research specific files. The licence situation is worth noting: the repository carries LICENSE.txt for the framework and a separate LICENSE-APPLICATIONS.txt, and the README points at the latter when it describes the SIGMA application, which is available under a research-only licence.

For a developer who prefers reading to watching, there is a short presentation in the Tech Minutes series, described as a shorter high-level overview of about 13 minutes, alongside the 30 minute video webinar with a 30 minute hands-on coding session.

Mixed reality and the HoloLens capture path

The release history shows where the effort went, and mixed reality is clearly the focus. The 0.17 beta release from April 2022 includes important updates to mixed reality support in Psi, including a set of tools for streaming data from a HoloLens 2 to a separate PC for data collection and export. The 0.18 beta from December 2022 continued to refine support for building mixed reality applications, and further evolved debugging and visualisation capabilities in PsiStudio.

The 0.19 beta from March 2024 is the newest tagged release, v0.19.100.1-beta, and it shipped alongside SIGMA, the Situated Interactive Guidance Monitoring and Assistance application, described as a baseline prototype and testbed system built on Psi and intended to accelerate research on mixed-reality task assistive agents.

That application is worth reading as a scope statement. It is a baseline and a testbed, not a product, and it is available under a research-only licence, so anyone building on it is working inside the research framing rather than shipping a product from it. The same release history also shows a MaskRCNN wrapper added, updates to the Azure Kinect components, and various runtime updates and bug fixes, which is a reasonable picture of where a platform with a large component surface spends its maintenance effort.

A framework that is still in beta

The version history is the most important fact about the project. The recent releases are v0.19.100.1-beta from 2024-03-14, v0.18.72.1-beta from 2022-12-07 and v0.17.52.1-beta from 2022-04-21, and the README itself uses the phrase beta version when describing each of them. Every tagged version carries the beta suffix, and the newest one is more than two years old, even though the repository is not archived and the last push was on 2026-10-19.

That combination, an active branch with no release since 2024, tells you the shape of the risk. The code is being worked on, and a consumer pinning to a NuGet package gets a beta from 2024, while anyone building from the master branch gets whatever the current state of the tree is, which the release notes and the wiki are the place to check.

The components list is also long, and the README points to a wiki page listing them, along with pages for building the codebase, using the framework via NuGet packages, a brief introduction, samples and the release notes. A framework this size with a component ecosystem is where the cost of an unmaintained release is concentrated, so the component list is the document to read before committing.

Where Psi is the wrong choice

Three situations point elsewhere. The first is a project that only needs to run one model over one stream. Psi is an infrastructure for joining several streams under latency constraints with visualisation and replay, and that machinery is a cost for a single-model pipeline where an ordinary library is enough.

The second is a team that needs a supported dependency chain. Every tagged release is a beta, the last one is from March 2024, and the framework mixes a permissive framework licence with a separate research-only licence for the applications directory. A production service that needs a versioned, supported dependency is not served by a beta line whose newest tag is two years old.

The third is a platform-sensitive deployment. The core runs on both Windows and Linux because it is built on .NET Standard, but some components and tools are specific to one of them, and the mixed reality work that the releases emphasise is tied to particular hardware such as the HoloLens 2 and the Azure Kinect. A deployment that has to be uniform across operating systems therefore has to check the component list rather than assume parity.

The alternative worth comparing against is a conventional dataflow or streaming library, where you assemble the same graph yourself and keep the component library as the only dependency. The difference in approach is that Psi makes the graph, the visualisation, the annotation and the replay part of one platform, which pays off when a research group iterates on a multimodal system across many experiments, and costs more when the requirement is a small pipeline that has to survive a maintenance cycle.

Editorial conclusion

Psi fits a research team building a multimodal interactive system, a social robot or a mixed reality assistant, that needs stream synchronisation, visualisation and a component ecosystem rather than a single inference model. It does not fit a team that wants a stable release to depend on, because the recent tags are all beta versions and the newest is v0.19.100.1-beta from 2024-03-14, and it does not fit a project that only needs one model on one stream. Before you adopt it, read the Brief Introduction tutorial and the release notes linked from the README, and check which components are Windows-only or Linux-only, since the core runs on both but some components and tools are specific to one.

Frequently asked questions

Official sources

1. Issues
2. microsoft/psi on GitHub
3. Project website
4. README
5. Releases