PIConGPU: a fully relativistic particle-in-cell code built to run on thousands of GPUs

What the PIC algorithm is and why it needs a supercomputer

PIConGPU is a fully relativistic, manycore, 3D3V particle-in-cell code. The Particle-in-Cell algorithm is described as a central tool in plasma physics, and it describes the dynamics of a plasma by computing the motion of electrons and ions in the plasma based on Maxwell's equations.

The 3D3V in the name is the standard decomposition: three spatial dimensions, three velocity components and one species. The code simulates individual charged particles, computes the electromagnetic fields they produce, and pushes the particles through those fields, which is a computation whose cost scales with the number of particles and which is embarrassingly parallel in exactly the way a GPU is built for.

That is the reason the project exists in the shape it does. The README makes the economic argument directly: as one of the supported compute platforms, GPUs provide a computational performance of several TFLOP/s at considerably lower investment and maintenance costs than multi CPU-based compute architectures of similar performance, and the latest high-performance systems are enhanced by accelerator hardware that boost their peak performance up to the multi-PFLOP/s level.

The scale claim that follows is the impressive one: with its performance and scalability to more than 18,000 GPUs, PIConGPU was one of the finalists of the 2013 Gordon Bell Prize.

The numerical schemes the code implements

The code implements various numerical schemes to solve the PIC cycle, and the list is the clearest way to understand what a researcher can configure.

For the fields there is a central or Yee-lattice, so the field grid can be laid out either with the fields at the cell centres or on the Yee lattice. The particle pushers solve the equation of motion for charged and neutral particles, and the named examples are the Boris pusher and the Vay-Pusher. The Maxwell field solvers include Yee's scheme and Lehe's scheme, both of which are cited to their papers.

Current deposition is described as rigorously charge conserving, with the Esirkepov scheme and EZ, Esirkepov meets ZigZag, given as the examples. That word rigorously is doing real work in plasma physics, since a deposition scheme that does not conserve charge introduces errors that grow over a long simulation and that are very hard to attribute.

The macro-particle form factors range from NGP at 0th order, through CIC at 1st, TSC at 2nd, PQS at 3rd, to PCS at 4th. These are the interpolation kernels that decide how a particle's charge is distributed onto the grid, and the order of the form factor trades accuracy against the number of grid interactions each particle needs.

Beyond the electromagnetic algorithm, the code is self-consistently coupled to classical radiation reaction, and to advanced field ionization methods, with the Keldysh and BSI schemes named. Radiation reaction matters in laser plasma work because the energy a particle loses to radiation changes its dynamics, and a self-consistent treatment is what separates a research code from a demonstration.

Diagnostics, output and the openPMD layer

Beyond the algorithm, the project developed a wide range of tools and diagnostics, and the list explains why this is usable as a working environment rather than a solver library.

There are online, far-field radiation diagnostics for coherent and incoherent radiation emitted by charged particles, which is the measurement that matters in laser-driven acceleration experiments. There are full restart and output capabilities via openPMD, including parallel HDF5, so a simulation that runs for hours can be checkpointed and resumed. There are 2D and 3D live view and diagnostics tools, which is what makes a run inspectable while it is going rather than after it ends. And there is a large selection of extensible online-plugins.

The openPMD choice is the strategically interesting one. It is an open standard for particle and mesh data, and adopting it means a simulation's output is readable by tools that know nothing about PIConGPU, which is what makes results shareable and checkable by people who are not the author of the code.

The repository layout reflects the same structure in the filesystem, with a src directory, an include directory, a lib directory, an etc directory with configuration, a buildsystem directory, a cmake directory, a test directory, a thirdParty directory, a share directory, a bin directory and a docs directory. There is also a libraryDependencies.png, a .zenodo.json for the archive record, a .gitlab-ci.yml since the primary development is on GitLab, and both a COPYING and a COPYING.LESSER file, which is the two licence files you would expect from the dual licensing arrangement.

The licence split between PIConGPU and PMacc

The licensing arrangement is unusual enough to be worth reading carefully before you build anything on it.

PIConGPU itself is licensed under the GPLv3+. Separately, you can develop your own particle-mesh algorithms based on the general library PMacc that is shipped alongside PIConGPU, and PMacc is dual licensed under both the GPLv3+ and the LGPLv3+. The detailed description is in the LICENSE.md file, and the repository carries both a COPYING file and a COPYING.LESSER file.

The practical effect is that the general-purpose part of the code is available under terms that let you build a closed-source tool on top of it, while the specific PIConGPU application remains copyleft. For a research group that wants to reuse the particle-mesh machinery in a code of their own, that is the licence that matters, and it is the reason to look at the license file rather than assume the whole tree is GPL.

The project is a scientific project with a citation requirement attached. If you present or publish scientific results that used PIConGPU, you should set a reference to show your support, and the up-to-date publication for the time of your publication should be taken from the REFERENCE.md file. There is also a community map to add yourself to, and a presentation slide intended for oral presentations that acknowledges the maintaining team.

For a codebase like this, a citation file that is expected to change between paper submissions is a small but characteristic detail.

Who develops it and how the project says to use it

PIConGPU is developed and maintained by the Computational Radiation Physics Group at the Institute for Radiation Physics at HZDR, in close collaboration with the Center for Information Services and High Performance Computing of the Technical University Dresden. The group is a member of the Dresden GPU Center of Excellence, which cooperates on a broad range of scientific GPU and manycore applications, workshops and teaching efforts.

The Users section of the README is the most surprising part of the documentation, and it is worth quoting closely. It emphasises that the developers are still actively developing PIConGPU at great speed and do, from time to time, break backwards compatibility. It tells you to stick to the latest release or use the dev branch containing the latest changes, and notes that the dev branch also contains a CHANGELOG.md with the latest changes and how to update your simulations, which should be read first before updating between two versions. It also notes that a git tag is added according to a version number for each release.

So the guidance to a user is: track the development, read the changelog, and expect to do some work when you move between versions. That is the opposite of a library's contract and entirely normal for a research code, but it means the code is a collaborator rather than a dependency.

The support model is also explicit and unusual. For any questions regarding the usage of PIConGPU, the README says please do not contact the developers and maintainers directly. Instead, open an issue on GitHub, and before posting, browse the documentation, the wiki and the issue tracker to see if the question has been answered. The default branch is dev, and the release history shows 0.8.0 in December 2024 with the atomic physics model FLYonPIC 2.0 and realistic laser pulse profiles, 0.7.0 in October 2023 with C++17 and PICMI support, and 0.6.0 in December 2021 with C++14, new solvers, openPMD I/O and HIP support.

Where PIConGPU is the wrong choice

The first limitation is that this is a research code, and it should be evaluated as one. The compatibility statement is in the README rather than hidden, and it is a commitment to breaking your build when the numerics change. A team that wants a stable dependency to put in a pipeline is looking at the wrong project, and the answer is a commercial simulation package or a maintained open source PIC code with a slower release rhythm.

The second is the build. There is no package manager artefact here, no binary distribution and no container in the repository listing. The installation notes live in a separate INSTALL.rst file, and the build system is a hand-assembled buildsystem directory with cmake files, pre-commit configuration and a clang-format and clang-tidy setup, which tells you the build is expected to be done from source with a toolchain you set up. For a physics group with a cluster and a sysadmin that is normal, and for everyone else it is a barrier.

The third is the domain. The particle-in-cell algorithm, Maxwell field solvers, radiation reaction and field ionization are the subject matter of plasma physics and laser plasma interaction. There is no general-purpose use of this code, and there is no abstraction layer that would let you use the Boris pusher for something unrelated, because the value is in the whole coupled scheme rather than in one routine.

The comparison with the alternatives is really a comparison of physics codes rather than of tools. The honest framing is that PIConGPU exists to make a specific class of experiment possible on hardware that a CPU cluster cannot reach, and a group whose experiment is in that class has little else to choose from.

Editorial conclusion

PIConGPU fits a plasma physics or laser research group that needs a relativistic electromagnetic particle-in-cell simulation at a scale only GPU hardware reaches, and that is prepared to build a research code with its own installation notes and changelog discipline. It does not fit a general engineering team looking for a library, and it is not a maintained application framework. Before building it, read INSTALL.rst and the CHANGELOG.md, because the project states plainly that it is developed at great speed and breaks backwards compatibility from time to time, and decide whether your own particle-mesh code should sit on PMacc, which is dual licensed and lets you develop your own algorithms under the LGPL terms.

Frequently asked questions

Official sources

1. ComputationalRadiationPhysics/picongpu on GitHub
2. Issues
3. Project website
4. README
5. Releases