Hysen LabsFastChat
An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.
FastChat, the platform behind Chatbot Arena
Over ten million chat requests served, more than seventy models, and one and a half million human votes. FastChat is the software under Chatbot Arena, and its repo is also the release home for Vicuna.
The numbers first
The platform runs Chatbot Arena at lmarena.ai, and the traffic is not small: more than ten million chat requests across seventy plus models. Visitors vote on side by side model outputs, and the project has collected over 1.5 million of those votes to feed an online Elo leaderboard. The repo is Python under Apache-2.0, with around 39,500 stars.
Training, serving, evaluating
The code splits into two halves. One half is training and evaluation code for the models the group built, Vicuna and MT-Bench among them. The other half is a distributed serving system that can host several models at once, with a web UI and RESTful APIs compatible with OpenAI clients. Together they cover the full loop the project cares about, from building a model to letting strangers judge it.
A release history you can read as a timeline
The news section doubles as a changelog. June 2023 brought MT-Bench. July 2023 shipped a dataset of 33k conversations with human preferences. August 2023 released Vicuna v1.5 on Llama 2 with 4K and 16K contexts, plus LongChat v1.5 with 32K. September 2023 added LMSYS-Chat-1M, a large real world conversation dataset. March 2024 closed the loop with the Chatbot Arena technical report.
Getting Vicuna to talk
Starting a chat with Vicuna triggers an automatic weight download from Hugging Face, cached in a folder under your home directory so it only happens once. Command options beyond the basics, including what to do when memory runs out, sit in the inference section of the docs. The intended path is a command line chat with no manual weight fetching along the way.
Community notes