ChatGLM-6B

ChatGLM-6B is an open bilingual dialogue model with 6.2 billion parameters, and its main selling point is that quantization brings it down to consumer graphics cards.

ChatGLM-6B is an open source dialogue model for Chinese and English, built on the General Language Model architecture. The README places the parameter count at 6.2 billion and says the training followed techniques similar to ChatGPT, tuned specifically toward Chinese question answering and dialogue. The repo is written in Python and released under Apache-2.0, and a larger lineup now exists: GLM-4-9B series models are open sourced, and an API platform hosts newer options like GLM-4, GLM-3-Turbo, and CogView-3.

About one trillion bilingual tokens went into pretraining, per the README, with supervised fine tuning, feedback-based self-improvement, and reinforcement learning from human feedback layered on top. The result, in the author's phrasing, is a model that produces answers reasonably aligned with human preference for its size. For downstream customization there is a P-Tuning v2 method that tunes only a small set of parameters.

The hardware story is the headline. With INT4 quantization the model needs about 6GB of VRAM for inference and 7GB for fine-tuning, which puts it inside consumer graphics card territory. FP16 without quantization sits at 13GB for inference and 14GB for fine-tuning, with INT8 between the two. After a couple of dialogue turns, memory usage lands near 10GB at 8-bit and 6GB at 4-bit. On macOS, an MPS backend works on Apple Silicon, though the half-precision model wants about 13GB of memory, and the INT4 kernel is CUDA-only so those builds fall back to CPU. Multiple GPUs are supported through the accelerate library.

The README is unusually direct about limits. A 6B model has a small capacity, so factual recall is shaky and logic problems in math or programming are not its strength. English output is weaker than Chinese, since most instruction data was Chinese, and the model can drift into mixed-language replies. It is also easy to mislead, with a weak sense of self that can be pushed off course.

The code is open under Apache-2.0, but the model weights follow a separate Model License. Academic use is fully open, and commercial use becomes free after registering through a questionnaire. The README also asks users not to put the model or derivatives toward harmful ends or unassessed services, and notes the team has not built any apps around it, from web to Android, iOS, or Windows.

• Source 1 · Project repository
• Source 2 · README