SPEAR drives Unreal Engine from Python instead of wrapping a fixed set of tasks

The design bet is that a general engine interface beats a purpose built simulator

The abstract states the problem in three words, and they are the same three words the whole project is built on. Existing photorealistic simulators suffer from limited generality, programmability and rendering speed, and SPEAR is the answer to all three. The mechanism is a Python library that connects to, and programmatically controls, any Unreal Engine application through a modular plugin architecture. The scale of that claim is what makes the project interesting, since SPEAR exposes over 14,000 unique engine functions to Python, which the documentation describes as an order-of-magnitude increase in programmable functionality over existing UE based simulators. The alternative designs, in the ecosystem SPEAR positions itself against, are simulators that implement a fixed set of tasks and scenarios. SPEAR does not implement tasks, it opens the engine. That is why the demonstration is six distinct embodied agents with different action spaces, and why they live inside sample projects the user already has.

Any reflected C++ function becomes a Python call

The programming model explains why the function count is so large. Any C++ function or variable that is visible to the engine reflection system, with SetActorScale3D and RootComponent given as examples, can be accessed as though it were a native Python function or attribute. That is the whole trick, and it is why the count is in the thousands rather than the dozens. The cost of that generality is synchronisation, and SPEAR handles it with an asynchronous variant provided for each function, with call_async.K2_GetComponentLocation named as an example, which avoids synchronising with the engine. For a research loop that issues thousands of calls per episode, that difference is between a workable simulation and an unusable one, and it is a design decision a task-focused simulator never has to face because it never exposes thousands of calls in the first place.

Work is expressed as transactions between begin_frame and end_frame

The unit of work in SPEAR is a graph of engine operations, and the graph is expressed as a transaction. The user specifies a transaction by defining a begin_frame context followed by an end_frame context, and within each context the graph of engine work is specified simply by implementing it as Python code. That is an unusual way to describe a dependency graph, and it is worth pausing on. Instead of declaring nodes and edges, you write ordinary Python inside two context managers, and the library works out what the work items are and how they depend on each other. The abstract describes the capability as an expressive high-level programming model that lets users specify complex graphs of UE work with arbitrary data dependencies among work items and execute those graphs deterministically within a single engine frame. Determinism within one frame is the property that matters for a training loop, since a frame that produces the same result every time is a frame you can put in a dataset.

The camera sensor writes photorealistic frames straight into NumPy

The sensor is the part of SPEAR that produces data rather than behaviour, and the numbers attached to it are the ones to evaluate. A single SPEAR instance can render 1920x1080 photorealistic beauty images directly into a user's NumPy array at 73 frames per second, described as an order of magnitude faster than existing UE plugins. The camera sensor is described as customizable, which is the point of a sensor abstraction, since the frame size, the field of view and the exposure are the parameters a researcher wants to vary across a dataset. The important structural detail is where the pixels land. Rendering into a NumPy array means the image never round trips through a file on disk or a network socket, which is what makes a throughput number like 73 frames per second meaningful for training. The repository carries a numpy_interop example directory, so the intended use is a Python process holding frames in memory.

Ground truth modalities are what the Hypersim dataset provides and this one also does

The second half of the sensor story is the part the documentation claims no existing UE based simulator offers. The camera sensor can render all of the image modalities in the Hypersim dataset, that is depths, surface normals, instance and semantic IDs, and a non-diffuse intrinsic image decomposition, as well as per surface material identifiers and physically based shading parameters. That list is the difference between a simulator that produces photographs and one that produces training data. Depth and semantic segmentation can be rendered from the scene, but per surface material identifiers and physically based shading parameters come from the engine's rendering internals, which is why they are only reachable if you are inside the engine rather than looking at it through a screenshot API. A non-diffuse intrinsic image decomposition separates the image into reflectance and shading, which is what a model for relighting or material estimation needs as a target. None of these need a second pass or a separate tool, they come out of the same sensor.

Getting a first instance running takes a config file and three calls

The starting point is short, and the shape of it tells you the design. You import the library, ask it for a configuration, create an instance and get the game object from it.

import spear

# create an instance representing a UE application; this will either launch a new UE
# application or connect to an existing one depending on how user_config.yaml is defined
config = spear.get_config(user_config_files=["user_config.yaml"])
instance = spear.Instance(config=config)
game = instance.get_game()

The comment on the config line carries the important detail, that the instance will either launch a new engine application or connect to an existing one depending on how user_config.yaml is defined. So the same code path serves an interactive debugging session against a running editor and a headless training run. The example directory list in the repository confirms the breadth of the shipped examples, with control_car, control_character, control_city_sample, control_content_samples, control_cropout_sample, control_editor, control_electric_dreams_sample, control_game_animation_sample, control_hillside_sample, control_metahumans_sample, control_simple_agent, control_stackobot_sample and debug_draw, alongside import_humoto_dataset, import_mixamo_dataset and import_stanford_dataset, mujoco_interop, editor_script_interop, enhanced_input, movie_render_queue, get_class_info and the notebook and storyboard examples.

The demonstrations are the argument that any engine project will work

The examples are chosen to make a specific point, that SPEAR does not require a specially built simulation. The six agents with distinct action spaces are a person and a car from CitySample, a flying robot from StackOBot, multiple agents in the resource collecting CropoutSample, and a person with parkour skills plus a quadruped robot from GameAnimationSample. Beyond the agents, the documentation shows procedural content generation being driven in the ElectricDreams sample, where a rock structure is translated and the scene harmonises around it, with the water adjusting and logs appearing, and time-of-day changes simulated by controlling the sky light orientation. There is also synchronized multi-view rendering of a human character in the MetaHumans sample, an interactive co-simulation with the MuJoCo physics simulator where the state of a MuJoCo scene is queried in real time and written into a corresponding UE scene, and an agentic application where a vision and language coding assistant edits a UE scene by iteratively writing SPEAR programs in response to text prompts.

Where SPEAR is the wrong choice

The first wrong case is a project without Unreal Engine. SPEAR is a plugin and a client library, so the engine is a prerequisite, and an engine installation with a C++ toolchain is a much bigger commitment than a Python package. The second wrong case is a project that only needs a fixed task scripted. If you need a robot to pick up a block in one room, a task-specific simulator gives you a smaller dependency and a faster loop, and SPEAR's generality becomes overhead. The third is a team that cannot accept a research codebase in the training path, because the asset licence and the code licence are deliberately different, with the code and assets released under an MIT License and a CC0 License respectively, and because the examples and the example directories show a project in active development rather than a stable release with a compatibility promise. The fourth is a project that needs deterministic rendering across machines, since the engine's own nondeterminism in rasterisation and lighting is inherited rather than removed, and SPEAR's determinism claim is about the execution of a work graph within a frame, not about pixel-identical output across a fleet. The fifth is hardware. A photorealistic sensor at 1920x1080 is a real GPU workload, and a lab without a capable GPU gets the library but not the throughput.

Editorial conclusion

Embodied AI researchers who need photorealistic rendering, a large action space and programmatic control of the scene in the same simulation loop are the people SPEAR is for, and the fact that the example agents run inside unmodified Epic sample projects is what makes it usable without building an engine project from scratch. It is the wrong tool for a team without Unreal Engine, since SPEAR is a plugin and a client rather than a standalone renderer, and it is the wrong tool for a project that only needs a fixed set of tasks scripted, because a general interface into the whole engine is more than such a project needs. Check three things before you commit: that the version of Unreal Engine you have matches the release tag you install, since the newest release is v1.1.0-ue58 and the tag names the engine version, that your graphics hardware and the engine build can sustain the sensor settings you intend to use, and that the ground truth modalities you need, such as material identifier values, are among the ones the camera sensor produces. The code is MIT licensed and the assets are CC0, and the last push was on 2026-10-19.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. Releases
5. spear-sim/spear on GitHub