sp_vision_25: a modular vision framework for a RoboMaster auto-aim system

What the team means by auto-aim, and what the code covers

This is the open source release of the auto-aim algorithm from the Tongji University SuperPower team for the 25 season of the RoboMaster competition. The project description gives the definition the team works with: software for automatic aiming and automatic fire control against moving armoured plate targets.

When the operator switches to auto-aim mode, the software takes over the gimbal, predicts the target's motion from its trajectory and solves the ballistics to steer the gimbal, and it also takes over the firing mechanism to decide when to shoot. The stated purpose is to raise the team's combat capability with a short kill time and a high hit rate.

The module decomposition is the part that generalises. The armoured plate detector and the target state estimator are the perception part, providing the enemy's movement information. The decision maker computes the aim position and makes the firing judgement from that information, and the controller executes the result. The figure in the project notes shows this split as a diagram of the whole auto-aim system.

The system is distributed software. The recogniser, the estimator and the decision maker run on a small computer with more compute, while the controller usually runs on an embedded microcontroller, with the split decided by real-time and stability considerations. The decoupling separates the vision team's responsibilities from the electronics team's.

What the repository contains is the vision team's part of that system: a multi-threaded camera driver, the electronics communication protocol and its implementation, image to quaternion alignment, camera intrinsic and hand-eye calibration programs, armoured plate recognition with both traditional and neural network implementations, armoured plate pose solving, coordinate transforms and yaw optimisation, a whole-vehicle estimator based on an extended Kalman filter, the aim point and firing decision logic, and separate test programs for each module.

One main function, many feature groups

The software architecture exists to solve a concrete problem, and the project notes explain it plainly. Different robot roles need several functions, but each function cannot be a separate program. An infantry robot needs auto-aim and buff reading, and both of those need images from the camera, but the camera can only be opened by one program, so the other one fails because the camera is busy.

The framework answers this by decomposing auto-aim, buff reading and the other functions into feature groups inside one framework, where each group contains multiple classes or functions such as the recogniser, the solver and the predictor. There is only one running program, the main function, and it fetches images from the camera and then selects the corresponding feature group to run based on the signal arriving from the electronics side, whether that is the auto-aim setting or the buff setting. Each robot role has its own execution logic, which means its own main function.

Beyond that, the framework provides a set of commonly used utility functions, with the stated aim of reducing the error rate and time cost of reinventing the same things.

The repository layout reflects the same modularity. There are directories for src, tasks, io, calibration, configs, tools, tests and assets, plus autostart.sh at the root, a CMakeLists.txt, a .clang-format file, and layout description files named buff_layout.xml and mpc_layout.xml, which are the schema descriptions for the two data types the electronics side exchanges.

The claim of being beginner friendly comes with an operational detail worth noting. The debugging setup in the project notes is a remote desktop session with NoMachine into the small computer on the robot, over a wireless network or a cable, paired with PlotJuggler to plot the data you care about, such as the gimbal yaw joint execution, the firing decision, and the angular velocity of the target being shot.

Building it on Ubuntu, and the dependency list

The documented environment is specific and worth quoting, because this code targets one hardware stack. The operating system is Ubuntu 22.04 and the compute platform is a NUC12WSKI7 with an i7-1260P and 16GB. The camera is a Hikrobot MV-CS016-10UC with the official 6mm lens, the lower board is a RoboMaster development board C type based on an STM32F407, and the IMU is the BMI088 built into that board. Communication is USB2CAN on the older setup and a MicroUSB virtual serial port on the new one. NoMachine and PlotJuggler are the auxiliary tools.

Beyond those three SDKs, the rest comes from the package manager:

    sudo apt install -y \
        git \
        g++ \
        cmake \
        can-utils \
        libopencv-dev \
        libfmt-dev \
        libeigen3-dev \
        libspdlog-dev \
        libyaml-cpp-dev \
        libusb-1.0-0-dev \
        nlohmann-json3-dev \
        openssh-server \
        screen

The build is a plain CMake configure and make, and the demo is a separate binary:

    cmake -B build
    make -C build/ -j`nproc`

    ./build/auto_aim_test

The project highlights mention that there is no ROS dependency, which means a new team member does not have to learn ROS to get started, though a ROS interface for communication with the navigation system is reserved for the sentry pursuit function.

The other highlight is the completeness of the workflow, covering development, compilation, debugging and deployment, and the modular architecture with per-module documentation, implementation and independent test programs. The earlier seasons show how that maturity accumulated: sp_vision_23 was written entirely in Python and proved the whole pipeline from recognition to firing, but ran at only 60FPS when the navigation was active, so sp_vision_24 was rewritten in C++ at about 100FPS and added parameter file loading, logging, offline detection with restart, a rosbag-like recording of timestamps, video and quaternions, and the per-module test programs that give the team its pre-match troubleshooting and post-match reproduction.

The trajectory planner, and the numbers the team reports

The season's main change is in the decision maker. Instead of the previous empirical, segmented decision logic, the team replaced it with a planner based on trajectory optimisation, and the claim is that this simplifies the code while giving better gimbal control and firing decisions. The theoretical framing is described as a trajectory-view theory of auto-aim, with a corresponding trajectory planning algorithm for auto-aim.

The data flow described in section 3.3 of the project notes sets out the pipeline end to end. The camera thread produces images and timestamps, and the lower board thread supplies the corresponding gimbal pose quaternion. The image goes to the recogniser, which returns the four corner pixel coordinates of an armoured plate and its pattern class. The estimator turns that into the target unit's motion state. The decision maker predicts the target's trajectory from that state, decides the best aim position and the best moment to fire, and forms a command sent to the lower board, where the controller and the actuator carry it out.

The reported results are specific enough to compare against your own setup. When tracking a single armoured plate at 5rad/s and 3m, the steady state error is under 0.01rad, and the algorithm has the ability to decelerate in advance when switching between plates. At the national competition, the highest hit rate recorded by the infantry robots running the algorithm was 39.6%, with a stated floor of 30% when performing normally. Kill time is reported separately: about 8 seconds against a 300HP enemy infantry robot at 2m with a 7rad/s rotation speed, and about 10 seconds at 14rad/s.

The detector also changed. The traditional image processing approach was replaced with a neural network four-point detection model opened by other teams, with the stated aim of raising the recall of the recogniser, while the decision maker moved to the trajectory planner. The traditional implementation is still in the repository, and the documentation lists both, along with a separate file for the supported expressions.

The deployment details that decide whether it runs unattended

The last part of the build instructions is the part that turns a working program into a robot that starts itself, and it is unusually specific.

Auto-start is set up with a desktop entry. You install screen, create the directory and the .desktop file, and write the entry with an absolute Exec path, which the documentation notes is required, plus a chmod to make the autostart.sh script executable. The Exec line points at /home/rm/Desktop/sp_vision_25/autostart.sh, which is the kind of detail that tells you the instructions were written from one specific robot rather than generalised.

The USB2CAN setup is optional and it is a udev rule file. The rule sets the CAN interface up on add, with the interface name, a can type and a bitrate of 1000000, for can0 and can1. The alternative and newer path is a serial connection, which needs its own rules file. That one takes the group dialout, then a udevadm query on the device to read its serial, idVendor and idProduct, and a rule that matches on those attributes and assigns a fixed symlink named gimbal. The rules are then reloaded and triggered, and the check is that /dev/gimbal appears as a symlink to the real device. The fixed symlink matters because the application should not have to know which tty the device enumerated as.

GPU inference is optional as well, and it installs a set of Intel graphics compiler and compute runtime packages, verifies the downloads against a checksum file, and installs them. The caveat attached to it is a real constraint: with GPU asynchronous inference the maximum display resolution is limited to 1920x1080 at 24Hz.

Where sp_vision_25 is the wrong choice

The first limitation is that this is a season's robot, not a library. There are no tagged releases, the version is not part of the contract, and the code is coupled to one hardware stack in ways that are not obvious from the outside: a specific camera SDK, a specific NUC, a specific development board with a built-in IMU, and a gimbal the controller half of the system expects. The last push was on 2026-08-13, so the repository is a snapshot of one team's 25 season work rather than a maintained package with a changelog and a support path.

The second is the domain. The code solves the problem of aiming at a moving armoured plate in a RoboMatch. The perception modules, the estimator, the ballistics solution and the firing decision all encode assumptions about that specific task, about a 2m to 3m engagement range, about the target's angular speed, and about the reaction time of a competition robot. There is no general purpose object detection here, and the trajectory planner is not a general motion planner.

The third is the depth of the hardware dependency. The build needs the MindVision or HikRobot SDK, OpenVINO, Ceres and the OpenCV and fmt and spdlog and yaml-cpp and Eigen stack, and the camera thread, the electronics thread and the estimator all assume they can talk to real devices. The per-module test programs soften this, and the previous season added a rosbag-like recording of timestamps, video and quaternions precisely so a problem could be reproduced after a match, but a team without that hardware can still only run so much of it.

The fourth is the competitive context. The reported hit rates and kill times come from national competition matches on the team's own robots, and the algorithm is a direct contribution to that team's performance. Anyone building on it is building on a system that was designed to win matches against other teams, and the documentation is written for people who already understand the domain.

The fifth is the missing maintenance. There is no issue tracker guidance, no release process, and the repository has no tags at all. For a team whose season starts in a few months, the value here is in reading and adapting, not in depending on it.

Editorial conclusion

sp_vision_25 fits a RoboMaster team starting a new season that wants a complete, working vision stack rather than a blank project, since it covers the camera driver, the armoured plate detector, the state estimator, the planner and the control protocol, each with its own test program, and the single-main architecture is the piece of design wisdom worth copying whatever else you take from it. It is the wrong choice if you need a general computer vision library, if your hardware differs from the documented NUC, Hikrobot camera and RoboMaster C board, or if you want a project with an active issue tracker and a support community. Before building on it, read the environment section and confirm the camera SDK, the IMU and the CAN or serial link match your hardware, run the auto_aim_test demo, and note that the last push was on 2026-08-13 with no tagged releases, so you are taking a snapshot of a season's code rather than a versioned library.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. TongjiSuperPower/sp_vision_25 on GitHub