simple-jev: classification from an open model's next-token logits

No classifier head, and no JSON from the model

Simple Jev is a project for structured classification and scoring with open models from Hugging Face, without training a separate classifier head. That last clause is the whole idea.

The usual way to build a classifier is to take a base model, put a classification head on it, and train it on labelled data. Simple Jev takes a model that already exists and asks it to score the options you care about, then turns those scores into structured output.

The mechanism is described precisely in the README. You send shared context and a set of questions. Simple Jev reads the model's next-token logits for each question and builds a JSON response containing choices, rubric scores, or truth and support judgments. And the detail that matters: the model does not generate a JSON completion. The server constructs the response from the scores.

That difference is the engineering point of the project. Asking a model to emit valid JSON with a specific schema means fighting the sampling process, retrying on parse errors, and accepting that a long answer can drift out of format. Reading logits over a known set of options removes the whole category of failure. A choice question becomes a comparison of the logits for each option, and the server decides which one won.

The audience is therefore engineers who already have a model and want structured output from it, rather than researchers looking for a new model. The current implementation runs locally with Hugging Face Transformers and PyTorch, and the demos, a playground and the documentation live at simple-jev.featherless.ai.

The common/ folder is the part other backends reuse

The repository layout is the clearest statement of the project's priorities, and it puts one directory first.

The top level holds .github/, .gitignore, LICENSE, README.md, and then RFDT/, common/, demos/, eval/, experiments/, hf-server/, imgs/, scripts/ and website/.

Three of those matter most. hf-server/ is the server implementation that the README's launch commands refer to, built on Hugging Face Transformers and PyTorch. eval/ is the evaluation framework, described in its own README. And common/ is the one the project description calls out explicitly: shared request validation, versioned prompt instructions, and response scoring live in the plain Python common/ folder so other inference implementations can use the same rules.

That separation is a real design decision. A server that owns the scoring logic and a scoring library that owns the rules are different things with different lifetimes. Putting the rules in plain Python, with the inference stack on top, means a second backend, or a different inference engine entirely, can produce the same answers from the same definitions. It also means the scoring behaviour is testable without a GPU, which is what makes the evaluation tooling in the next section possible.

The rest of the tree is research scaffolding rather than library code. demos/ and website/ hold the public-facing pieces, experiments/ holds whatever the authors were trying, RFDT/ is a component the README does not explain, imgs/ holds images, and scripts/ holds helper scripts.

The licence is Apache-2.0, which is a permissive choice that makes it straightforward to use the rules in a product.

A public demo with no key, two requests per second and 2k tokens

There is a public demo API that needs no login, no API key and no authentication at all. That makes it the fastest way to see what the response actually looks like before you run anything locally.

The limits are stated in the same breath. The demo has a 2k-token context limit and is rate limited to 2 requests per second. For production deployments the README points at Featherless paid plans with higher limits.

The first call lists the available models:

curl https://simple-jev-demo-api.featherless.ai/v1/models

The demo already serves Gemma as featherless-ai/gemma-4-26B-A4B-classifier, and you can post to the classifier endpoint directly, or use any other model id the list returns:

curl https://simple-jev-demo-api.featherless.ai/v1/classifier \
  -H 'Content-Type: application/json' \
  --data-binary @- <<'JSON'
{
  "model": "featherless-ai/gemma-4-26B-A4B-classifier",
  "state": "Mia owns a red bicycle.",
  "questions": {
    "color": {
      "type": "choice",
      "instructions": "What color is Mia's bicycle?",
      "criteria": {"red": null, "blue": null}
    }
  }
}
JSON

The request shape is the same one you would use against your own server. There is a model, a state string holding the shared context, and a questions object keyed by question name. Each question carries a type, here choice, an instructions string, and a criteria map that lists the options with null values as placeholders.

That the options are null rather than carrying weights or examples is a small detail that tells you where the design is heading: the question defines the space, the model scores it, and the server returns the result.

The limits are startup flags, and they are tight

The most important operational detail in the README is a warning that is easy to miss: the question and context limits are server startup flags, not fields in a classification request.

That means you cannot widen them per call. You restart the server with different values, and the defaults apply to everyone who hits that process.

There are three of them. The first is --max-request-branches, the number of questions per HTTP request, defaulting to 100, with 256 described as the schema maximum and a note that higher values cannot bypass that maximum. Each question uses one scoring branch.

The second is --max-model-len, the model input length, defaulting to 16384 tokens per complete rendered question branch. The README is careful about what that counts, and the detail matters: it is tokens per branch including the state and history, the system instructions, the options, the template overhead and any policy repetition, not characters and not generated tokens. So a long shared context eats the budget of every question in the request rather than being paid once.

The third is --max-choice-options, the number of choices per choice question, configurable from 2 to 255 and defaulting to 255. It is per question, not the number of questions, and the score itself remains limited to 50 levels with Noul unchanged.

The README then shows the combination for a large model, allowing 256 questions per request, 32K input tokens per branch and 255 choices per question, and notes that a recognised Qwen 27B configuration auto-selects the binary examples format with no format flag, giving the example of a request with three questions containing 255 choices.

For an evaluation harness that has to sweep a lot of prompts, these numbers are the design envelope, and they are the first thing to check against your workload.

Running the server locally on a CPU first

The setup uses Python 3.12 or newer, with the commands shown using Python 3.13. The sequence is a clone, a virtual environment and an editable install of the server package including the shared common modules.

The first launch example starts a small model on a CPU, which is the right way to check the plumbing before you commit to a GPU:

python hf-server/hf_server.py \
  --model Qwen/Qwen3.5-0.8B \
  --device cpu --dtype float32 \
  --classifier-prompt-policy baseline \
  --max-model-len 4096 \
  --max-batch-size 4 --max-batch-tokens 4096

The flag --classifier-prompt-policy baseline is the one to notice. The prompt instructions are versioned, and the policy selects which version the server renders, which is how the project keeps a scoring run reproducible when the prompt template changes.

Two larger examples follow. One runs Gemma 4 MoE on an NVIDIA GPU with BF16, with a longer model length and a larger batch budget. The README is careful about that example: it says to allow memory for the full model weights, the KV cache and the inference buffers, and that sparse expert activation does not mean only the active experts occupy memory. It also says plainly that this is a launch example and not a verified full-size Gemma benchmark, which is a good habit for a project to keep.

The third example uses the Laya Typed Decisions backend with a specialised checkpoint, its own subfolder, and --rope-factor 2 enabling experimental 2x linear RoPE interpolation. The README states that this enables longer inputs but does not establish accuracy or calibration beyond the checkpoint's training length, and that omitting the flag retains the native behaviour.

The server listens on http://127.0.0.1:8000, exposes a health endpoint and serves interactive documentation at the /docs path, and after installation the simple-jev command and the python -m hf_server module accept the same arguments as the script.

An evaluation framework with matched-metric comparisons

The eval/ directory is described as an evaluation framework, and the feature list is specific enough to show what the project thinks an evaluation needs.

It includes native JevBench scoring, text and decision suites as well as vision suites, dataset preparation, resumable HTTP execution, raw-response audits, and matched-metric comparisons. That last group is the interesting one for anyone evaluating a scoring system.

A raw-response audit keeps the model's original output next to the parsed result, and a matched-metric comparison lines up two systems on the same metric rather than on the same headline number. Together they address the two failure modes of a classifier evaluation: silently dropping or mis-parsing responses, and comparing systems whose numbers are not measured the same way.

Three commands are documented:

python3 eval/run.py --preset quick --list
python3 eval/run.py --preset full --list
python3 -m unittest discover -s eval -p 'test_*.py'

The quick preset lists 477 development and selection decisions, and the full preset lists the frozen text and vision selections. The third is the offline check, run with the standard library unittest discovery, and the README notes that the client needs no GPU.

The last point is worth pausing on. The evaluation client talks to a classifier endpoint over HTTP, so the same evaluation can run against the public demo, against a local server, or against a hosted deployment, and the heavy model runs on the server side.

What is not bundled is as important as what is. Large datasets, model weights and raw evaluation runs are not included, and the README points at the reproduction instructions in eval/README.md for portable presets and full reproduction.

A young project with a public surface and no releases

The project state is straightforward. The last push to the repository was on 2026-10-07, there are no GitHub releases, and the version numbers that matter are the model ids and the request schema rather than a package version, since the deployable unit is a Python package you install from the repository with pip install -e.

The public surface is larger than the code, though. There is a public demo API with no authentication, a demos and playground site, a website directory in the repository, and a Discord-style community implied by the Featherless platform. That is a deliberate choice: the barrier to trying the approach is a single curl command.

The split between the demo and production is the part to keep in mind. The demo is free and unauthenticated, which is generous, and it is also rate limited to two requests per second with a 2k token context limit. Anyone whose evaluation harness needs thousands of decisions will hit that wall quickly, which is presumably the point of the paid plans.

On the licence, Apache-2.0 is permissive and includes a patent grant, which matters if you intend to build a product on top of the shared rules in common/. The repository also carries a CODE_OF_CONDUCT.md and a LICENSE file at the top level.

The honest summary is that this is a young project with an unusually clear mechanism. It does not ask a model to produce structured output, it reads the logits and constructs the output itself, and it separates the rules from the inference stack so that the rules can be tested and reused. The cost is that the shape of the question is fixed, the limits are fixed at startup, and the interesting engineering has to happen in the common/ folder.

Editorial conclusion

Simple Jev suits teams that need labelled or scored output from a model they already run, and that would rather constrain an open model with a rubric than collect training data and fine-tune a classifier. It does not suit a project whose question does not fit the choice, rubric or truth and support formats, because the response is constructed from logits over a fixed set of options rather than generated, and it does not suit anyone expecting the public demo to be a production dependency, since it carries a 2k token context limit and a rate limit of two requests per second. Check three things before you rely on it: which of the startup limits your workload needs, since the request branch count, the per branch token budget and the choice count are all fixed when the server starts, that your model is compatible with the prompt policy, and that the shared common/ rules match the behaviour you would have written yourself. The last push to the repository was on 2026-10-07 and the project publishes no GitHub releases.

Frequently asked questions

Official sources

1. featherless-ai/simple-jev on GitHub
2. Issues
3. License: Apache-2.0
4. README