DataFlex: sample selection, domain mixing and reweighting inside the LLaMA-Factory loop

The problem DataFlex addresses: data schedules that die in a private fork

Most data-centric training papers ship as a repository that reimplements the training loop just enough to demonstrate the method. LESS has an official repo, NICE has one, DOREMI and ODM have one each, and the README marks several of them with a warning symbol rather than a clean check. Reproducing a comparison across four of those methods means reconciling four codebases, four dependency sets and four ideas of what a training step looks like. DataFlex's stated goal is to fold those into one framework on top of LLaMA-Factory so that selection, mixture and reweighting are configuration choices rather than separate installations. The audience is therefore narrow and identifiable: researchers and engineers who already train with LLaMA-Factory and want to vary the data schedule as an experimental variable. It is not aimed at teams who just want a working fine-tune.

Three intervention points: selector, mixer and reweighter

The README splits the framework into three roles. A selector picks which training samples participate, with methods grouped as gradient-based (LESS, NICE), loss-based (Loss, Delta Loss), distribution-based (NEAR, TSDS) and the two baselines Static and Random. A mixer adjusts the ratio of data drawn from different domains during training, with DOREMI as the offline method and ODM as the online one. A reweighter adjusts sample weights during backpropagation, with Loss Reweighting and Joint-Update-Aware Reweighting listed. The README's own table adds a column that matters more than the categories: whether the method requires the model in the loop. Gradient-based selectors and both mixers do; NEAR, TSDS, Static and Random do not. That distinction predicts your compute bill far better than the method name does, because a model-in-the-loop method needs forward or backward passes over candidate data to make its decisions. The repository also documents a registry system and base class interfaces in skills/how_to_add_algorithm.md, which is where the extension path lives if none of the shipped methods fits.

Installing DataFlex without breaking the torch pin

The README is explicit that torch goes in first, so pip does not resolve a different build and then have to replace it:

pip install --index-url https://download.pytorch.org/whl/cu124 torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 python -c "import torch; print(torch.__version__, torch.cuda.is_available())"

Then either pip install dataflex, or clone the repository and run pip install -e . for development. DeepSpeed is not a core dependency. Every config under examples/deepspeed needs it, and it arrives through an extra: pip install "dataflex[deepspeed]", or pip install -e ".[deepspeed]" from a source checkout. The LESS selector needs TRAK, which is a third extra, pip install dataflex[less]. A launch looks like the LLaMA-Factory equivalent, for example dataflex-cli train examples/train_lora/selectors/less.yaml. The README states that unlike vanilla LLaMA-Factory, the YAML must also carry DataFlex-specific parameters, and points at DataFlex-Doc for the key names. That documentation gap is real: the README tells you the keys exist without listing them, so the first configuration is a documentation-reading exercise rather than a copy-paste.

The version constraints are the adoption cost

Python 3.11 or later and LlamaFactory 0.9.5 or later are required, with LlamaFactory installed automatically alongside the other core dependencies. Transformers support spans 4.55 through 5.6, but the newer model families named in the README, Qwen3.5 and Gemma 4, need 5.5 or later. For train_from_scratch under DeepSpeed ZeRO-3 the README recommends transformers 5.3 or later. The deepspeed extra is deliberately held below 0.17 because 0.17 and later fail to import against the pinned torch; the README notes you can lift that cap on a newer torch, which means the constraint is a consequence of the torch pin rather than a DeepSpeed defect. ZeRO-3 gradient computation is listed as a 2026-03-17 addition, so the combination of ZeRO-3 with the newer transformers recommendation is recent enough that it deserves a smoke test on your own cluster before a long run. This is a stack where the dependency graph, not the algorithm, is the thing most likely to consume your first day.

Where DataFlex is the wrong tool

If your training run already converges and your data is fixed, DataFlex adds decision overhead for no return. Every model-in-the-loop method spends compute on scoring candidate samples or estimating domain weights, and that compute competes with the training run itself. The README's selector table is the honest guide here: if you pick NEAR, TSDS, Static or Random, you are in the no-model-in-the-loop column and the framework is mostly acting as a data-pipeline shim you could have written yourself. The framework is also a poor fit if you are not on LLaMA-Factory. The whole value proposition is drop-in compatibility with that training loop, and the README describes DataFlex as built on top of it; using DataFlex with a different trainer discards the integration that justifies the dependency pinning. Finally, if your environment is locked to a torch newer than 2.6.0 in a way you cannot change, the deepspeed cap becomes a constraint you have to resolve deliberately rather than inherit.

How this differs from plain LLaMA-Factory, and from LESS alone

Plain LLaMA-Factory trains on a static dataset you hand it. It has no concept of a selector that changes the sample set between epochs, a mixer that shifts domain ratios mid-run, or a reweighter that modifies per-sample gradients. DataFlex inserts those three as pluggable components with a shared registry and base classes, so the difference is not a feature but a control surface. Compared with adopting LESS on its own, the trade is breadth against depth. LESS is one gradient-based selector with its own official repository; DataFlex ships LESS plus NICE, the loss-based and distribution-based selectors, both mixture methods and both reweighting methods, behind one CLI and one config format. If LESS is the only method you will ever run, the single-purpose repository has fewer moving parts and no obligation to keep transformers, DeepSpeed and TRAK mutually compatible. If you intend to compare methods, the single-purpose path means repeating the integration work per method.

Licence, maintenance and what the release history shows

DataFlex is Apache-2.0, which permits commercial use and modification provided the licence and notices are preserved; that is a description of the licence terms, not legal advice, and the bundled LLaMA-Factory and any optional extras such as TRAK carry their own terms you should check separately. On maintenance, the evidence in the material is a v1.0.0 release dated 2026-04-16, a last push dated 2026-09-10, and a news entry from 2026-03-17 adding ZeRO-3 gradient computation. The gap between the v1.0.0 tag and the most recent push suggests active work between releases rather than a frozen tag, which for a research framework usually means you should track a commit rather than assume the release artifact is current. The README also states that several upstream methods are integrated from repositories it marks with warnings, so bug reports may need to be routed upstream rather than to DataFlex. Budget for that: a defect in a selector's scoring logic is not necessarily fixable in this repository.

Editorial conclusion

Adopt DataFlex if you already run LLaMA-Factory and want to compare LESS, NICE, DOREMI or ODM without reimplementing each paper against a different training harness. Do not adopt it if you need a stable dependency set, since the README pins torch 2.6.0 with cu124 and holds the deepspeed extra below 0.17 specifically because 0.17 or later fails to import on that torch. Before committing, verify two things on your own hardware: that your target model works on the transformers range you install, and that the LESS path actually imports TRAK, which lives in a separate dataflex[less] extra and is not pulled in by the base install.

Official sources

1. License: Apache-2.0
2. OpenDCAI/DataFlex on GitHub
3. Project website
4. README
5. Releases