Taxi: SystemVerilog building blocks for FPGA data transport

A component library, with two large systems living inside it

Taxi is described as a set of performant, easy to use building blocks in modern SystemVerilog, for data transport and interfacing. The interfaces split cleanly in two. Internally the components speak AXI, AXI stream and APB, which is how blocks inside one FPGA talk to each other. Externally they cover Ethernet, PCI express, UART and I2C, which is how the design talks to the outside world. That division is what makes a library like this useful, because the components are built to fit together. The library is also the home of Corundum, a NIC and platform for in-network compute, Zircon, an IP stack, and XFCP. So the repository is two things at once: a grab bag of reusable components and the home of two substantial systems, and a reader has to decide which of those they are looking at. The documentation is at docs.fpga.taxi, the project site is fpga.taxi, and the community runs on Zulip at corundum.zulipchat.com.

Cocotb and Verilator are how the components are trusted

The building blocks are accompanied by testbenches and simulation models that use Cocotb and Verilator, and the repository carries a regression test workflow with its own badge. This is the part that decides whether the library is usable, because an AXI stream component that has never been driven by a test is a component that will cost you a week. Cocotb is a way of writing testbenches in Python against a simulation, and Verilator is the simulator underneath, so the testing story is a Python one rather than a SystemVerilog one, which is a large deal for a team whose verification skills are in Python rather than in UVM. Two other files in the tree point the same way: a tox.ini for running the environments, and a .test_durations file, which is the sort of bookkeeping that only matters when you are running a large suite repeatedly and want the slow tests scheduled sensibly. The README is also explicit that the library is currently under development and that more components will be added as they are written, so treat the component list as a moving target.

Corundum is a NIC aimed at in-network compute

Corundum is an open source, high performance FPGA based NIC and platform for in-network compute, and the feature list reads like a requirements document for a serious card. There is a high performance datapath, 10G, 25G and 100G Ethernet, PCI express generation 3 and later, a custom high performance tightly integrated PCIe DMA engine, and more than a thousand transmit, receive, completion and event queues. On top of that sit scatter and gather DMA, MSI and MSI-X interrupts, per port transmit scheduling, flow hashing, RSS, checksum offloading and native IEEE 1588 PTP timestamping. Two pieces of software come with it: a Linux driver that integrates with the Linux networking stack, and a DPDK PMD. The development story is the simulation framework, which covers the entire system, from a model of the driver and the PCI express interface on the host side through to the Ethernet interfaces on the network side. Being able to reproduce a full host and network path in simulation is what makes the performance work tractable, and it is why the project invests in it.

Four Corundum variants for different optimisation points

Several variants of Corundum are planned, sharing the same host interface and device driver while targeting different optimisation points, and the naming is worth unpacking because it tells you which one to look at. corundum-micro is size optimised for applications such as SoCs and low bandwidth NICs, supporting several ports at 1 Gbps up to 10 or 25 Gbps. corundum-lite is the middle of the road design, supporting multiple ports at 10G or 25G or one port at 100G, with aggregate throughput of around 100 Gbps. corundum-ng targets high performance packet processing with deep pipelines and segmented internal interfaces, operating up to 400 Gbps aggregate. corundum-proto is a simplified design with a simplified driver, intended for educational purposes only. The honest caveat is in the README: Corundum is still under active development and may not be ready for production use, with more functionality and improvements to performance and flexibility to come. Planned features include SR-IOV, AF_XDP, white rabbit and IEEE 1588 high availability, and Zircon stack integration.

Zircon parses headers, then hands you what you asked for

Zircon is the IP stack, and it implements IPv4, IPv6 and UDP from 1 Gbps to 100 Gbps. The design detail that matters is what happens to a packet on the way through. The stack handles parsing and deparsing of packet headers, inserts and removes VLAN tags, computes and verifies header and payload checksums, matches receive packet fields against configured rules, and multiplexes between multiple internal application interfaces. That last part is where the design earns its keep: the stack can be configured to pass packets to application logic unmodified, with VLAN tags stripped, with simplified headers, or with only the payloads. A design that wants raw payload bytes for its own routing does not have to undo the stack's work. As with Corundum, the README states that Zircon is under active development and may not be ready for production use, and the planned features are TCP, RDMA and integration with Corundum. In other words, the transport protocols most applications want are the ones still to come.

Ethernet MAC and PCS variants, with two hard limits

The Ethernet side of the library has several MAC and PCS variants covering link rates from 10 Mbps to 25 Gbps. The MAC modules support LFC and PFC pause frames, PTP timestamping, frame length enforcement, FCS computation and verification, and statistics reporting, with PTP TD leaf clocks and statistics collection components integrated and configurable through instance parameters. Low speed operation is handled by MII, GMII and RGMII wrappers for an external PHY chip, and there are device specific transceiver instances for 1000BASE-X and 10GBASE-R and 25GBASE-R. Logic is available for a 10G and 25G MAC, the matching PCS, and a fused MAC and PCS with reduced latency and resource consumption. Two limits are stated plainly. The 100G wrapper for UltraScale and UltraScale+ uses the hard CMAC and GTY transceivers, currently only at 100 Gbps, and RS-FEC is enabled only on UltraScale+, because the CMACE4 provides it in hard logic while the UltraScale CMACE3 needs separately licensed soft logic. The 10G and 25G wrapper spans 7-series, UltraScale and UltraScale+ with GTX, GTH and GTY transceivers, a 32 or 64 bit datapath chosen with the DATA_W parameter, USXGMII support for NBASE-T PHYs and SFP modules with dynamic detection, and TCL scripts for generating the GT cores. The low latency mode, set with CFG_LOW_LATENCY, trades a slightly higher clock frequency and more resources for a few nanoseconds, and the README advises against it unless you need it.

The licence asks for your whole design, and there is a paid alternative

Taxi is provided by FPGA Ninja, LLC under either the CERN Open Hardware Licence Version 2 Strongly Reciprocal or a paid commercial licence, with commercial use arranged through [email protected]. Some components may carry less restrictive terms, and example designs are given as an example of that. The obligation to understand is specific: under the strongly reciprocal CERN OHL, you must provide the source code of the entire digital design upon request, including all modifications, extensions and customisations, so that the design can be rebuilt. That is a wider obligation than a normal open source licence, and for a commercial product built on top of these components it may be the deal breaker. The README points out the way out, a commercial licence without that requirement, and says the fees support the continued development and maintenance of the project. One more procedural detail closes the loop: to facilitate the dual licence model, contributions to the project can only be accepted under a contributor licence agreement, so if you want your own work upstream you sign over the rights the maintainer needs to offer both licences.

Editorial conclusion

Taxi is a good fit for an FPGA team that has to build a custom datapath and wants tested, reusable transport components plus a fast path to a real NIC, and the Cocotb and Verilator testbenches are the reason to prefer it over writing your own. It is the wrong choice if you need a production-ready network stack today, since both Corundum and Zircon are still under active development, and it is the wrong choice if the source disclosure obligation of the CERN OHL is incompatible with your product. Before you commit, read the licence terms against your own design, talk to [email protected] if you need a commercial licence, and scope your plan around the fact that TCP and RDMA in Zircon are planned rather than present.

Frequently asked questions

Official sources

1. fpganinja/taxi on GitHub
2. Issues
3. License: CERN-OHL-S-2.0
4. Project website
5. README