Hysen Labsbumblecore
An LLM training framework built from the ground up, featuring a custom BumbleBee architecture and end-to-end support for multiple open-source models across Pretraining → SFT → RLHF/DPO.
bumblecore: an LLM training framework built by hand
BumbleCore rebuilds the whole LLM training stack from scratch, with no high level Trainer library in sight, a BumbleBee architecture inspired by Qwen2.5, and DeepSpeed doing the heavy lifting.
Why build the training loop by hand?
The pitch is deliberate. BumbleCore does not wrap a high level Trainer library. Custom data loaders, manual distributed setup with deep DeepSpeed integration, a forward, backward, and update flow you control, flexible losses with multi task support, and hand built inference with Top-p and Top-k sampling plus KV Cache. The README argues the payoff is understanding: when you implement every line yourself, debugging and experimentation get easier.
What does the BumbleBee architecture give you?
The built in BumbleBee architecture, inspired by the Qwen2.5 design, handles configuration. You scale parameters from small experimental models up to production sized ones, adjusting Transformer layers, attention heads, and hidden dimensions as needed, with pluggable activation functions, normalization, and attention. The README claims you can configure a model and start training in minutes.
How far does the pipeline go?
The pipeline covers pretraining, SFT, and DPO, and the README lists the Qwen and LLaMA series as fully tested across all three stages. Data comes in JSON or JSONL with automatic recognition, using Alpaca, ShareGPT, or OpenAI style Messages formats, and DPO data carries chosen and rejected pairs. A quick start walks through SFT with shell scripts, and there is a full three stage tutorial, LoRA weight merging, and inference through YAML or command line flags.
Editorial conclusion
BumbleCore is a from scratch LLM training framework: manual internals, DeepSpeed for scale, and the Qwen2.5 inspired BumbleBee architecture, verified on Qwen and LLaMA across pretraining, SFT, and DPO.
Community notes