Promptify: Pydantic-Typed NLP Tasks Behind a LiteLLM Backend

The problem Promptify addresses: task-shaped output instead of chat strings

Calling a language model for an NLP task usually means three separate jobs: writing a prompt that constrains the output format, parsing whatever comes back, and validating it. Promptify collapses those into a class per task. The README describes it as a "Task-based NLP engine with Pydantic structured outputs, built-in evaluation, and LiteLLM as the universal LLM backend", and the quick tour shows the shape of that claim. NER returns a NERResult containing Entity objects with text and label fields. Classify returns a Classification with label and confidence. QA returns an Answer with answer, evidence and confidence.

The intended audience is Python engineers who need structured extraction or classification and do not want to maintain a prompt template per model vendor. The README's own framing is "scikit-learn for LLM-powered NLP", which sets an expectation of a uniform estimator-style interface. That expectation mostly holds for the built-in task classes. It holds less well for the parts of a scikit-learn workflow that people lean on later, such as reproducible fitting and held-out scoring, because the underlying model is remote and the prompt text is not exposed as a versioned object in the documented API.

How a task call flows: class, schema, LiteLLM, safe parser

Each task class carries an output schema. NER maps to NERResult, Classify to Classification or MultiLabelResult when multi_label=True is passed, Summarize to Summary, ExtractRelations and ExtractTable to ExtractionResult, GenerateSQL to SQLQuery. The Task class takes output_schema as a constructor argument, so a custom Pydantic BaseModel can stand in for any of the built-ins. That is the mechanism that makes the type annotations real rather than decorative.

Provider selection happens through the model string. The README gives three examples in one block: model="gpt-4o-mini", model="claude-sonnet-4-20250514", and model="ollama/llama3". Nothing else changes between them. LiteLLM resolves the prefix and handles the transport, which is why the feature list claims OpenAI, Anthropic, Google, Ollama, Azure and "100+ more".

The parsing layer is worth noting because it is where structured-output libraries usually break. The README advertises a "Safe parser" with "fallback JSON completion for providers without native structured outputs (no eval())". That phrasing implies two paths: providers with native structured output return a conforming object, and providers without it get a completion pass that fills in the JSON. The absence of eval() matters for anyone who has read the older generation of prompt libraries, where string-to-dict conversion via eval was common.

Batch and async sit on top of the same call path. ner.batch([...], max_concurrent=10) fans out with async concurrency, and await ner.acall("Patient has diabetes") is the single-item async form. The README does not document retry behaviour, backoff, or what happens to a batch when one item fails validation, so treat max_concurrent as a throughput knob whose failure semantics you will need to discover from the source.

Getting it running: install, extras, and the two config surfaces

Installation is a single pip command, with Python 3.9 or newer required. The README lists two forms:

pip install promptify

pip install git+https://github.com/promptslab/Promptify.git

The second form installs from the default branch rather than a tagged release. No releases were retrieved for this review, so if you need a pinned version, the PyPI path is the one with a version number attached.

Evaluation metrics live behind an extra:

pip install promptify[eval]

That is the only optional dependency group the README names, and it is the switch that enables the metrics listed later: precision, recall, F1, accuracy, exact match and ROUGE.

Configuration is deliberately thin. There are no documented config files and no documented environment variable names. The two surfaces you control are the constructor arguments (model, domain, labels, multi_label, output_schema, instruction) and whatever credentials LiteLLM reads from the environment for the provider you selected. The domain argument is the one with the most leverage over output quality: domain="medical" in the NER example is described as passing context for context-aware prompts. The README does not specify whether domain is injected as a system message, a prompt prefix, or a label vocabulary, and that difference changes how much it actually helps. If domain behaviour matters to your use case, read the prompt construction code before relying on it.

What the evaluation module measures, and what it does not

The eval entry point is a single function:

from promptify.eval import evaluate scores = evaluate(task=ner, dataset=labeled_data, metrics=["precision", "recall", "f1"])

The task argument is a task instance, so the same object you call in production is the object being scored. That is a reasonable design: it avoids the common drift where the evaluation harness builds prompts differently from the serving path. The dataset is described as labeled_data, and the metrics list is explicit, which means scoring is opt-in rather than a default suite.

The limitation is in the metric set. Precision, recall, F1, accuracy, exact match and ROUGE are string and set comparison metrics. They score the parsed output against a reference. They do not measure whether the confidence field on Classification or Answer is calibrated, and the README gives no indication that calibration is checked anywhere. If you plan to threshold on confidence=0.95 or confidence=0.98 as the examples show, the evaluation module will not tell you whether those numbers mean anything. That is a gap worth knowing about before you build routing logic on top of them.

There is also no documented mechanism for storing a dataset alongside a prompt version, or for comparing two prompt variants against the same labeled set. The function signature takes one task and one dataset, so A/B comparison of prompt variants is something you would assemble yourself.

The provider abstraction is a string, and that is the main constraint

LiteLLM as the backend is the design decision that shapes everything else. It buys provider portability at the cost of a thin contract: the library knows the model name and the output schema, and delegates everything about how the request is formed to LiteLLM. The README does not document per-provider prompt overrides, temperature, max tokens, or any sampling parameter. If a provider needs a different prompt phrasing to hit the same accuracy, the documented API gives you no place to put that.

The fallback parser is the other side of the same coin. Providers with native structured outputs get a constrained generation path; providers without it get JSON completion after the fact. Those two paths can produce different error rates on the same input, and the README does not quantify or characterise the difference. For a local model behind ollama/llama3, the fallback path is the one you will be on, and its reliability depends on the model's instruction following rather than on any grammar constraint the library enforces.

Cost tracking is exposed through get_cost_summary(), which the feature list ties to token usage. That is useful for batch runs, but the README does not state whether the accounting is per task instance, per batch, or process-wide, so the scope of the summary is something to confirm in code before you use it for budgeting.

Where Promptify is the wrong tool

Three cases stand out from the documented material.

First, anything that must run without a network call to a model provider. Every task class takes a model string, and the examples all resolve to either a hosted API or a local server. There is no documented offline or cached mode, so a pipeline that must produce identical output on every run without a live model is outside what the README describes.

Second, work where the prompt itself is the deliverable. The repository topics include prompt-versioning and prompt-tuning, and the project description mentions prompt versioning, but the README's API surface has no prompt object, no version identifier, and no way to pin a prompt revision to a result. The Task class takes an instruction string and an output schema. Nothing in the documented interface records which instruction produced which output. If your requirement is an auditable prompt registry, this library does not provide it, whatever the topic tags suggest.

Third, high-volume batch jobs where a single malformed response should not be silently absorbed. The safe parser exists to make JSON parse, and the README does not describe what happens when the fallback completion still fails to match the schema. A parser that repairs output is the right choice for exploratory work and a questionable one for pipelines where a wrong entity label is worse than a raised exception. The README does not document a strict mode, so you would need to detect that case yourself.

The alternative: raw provider SDKs plus instructor or Outlines

The closest comparison is not another task library but the combination of a provider SDK with a schema-binding layer such as instructor or Outlines, or with the provider's own structured-output mode. The difference is where the abstraction sits.

Promptify abstracts at the task level. You ask for NER and get a NERResult; the prompt that produces it is the library's concern, and domain is the only documented knob for steering it. A schema-binding library abstracts at the call level: you write the prompt, you declare the Pydantic model, and the library constrains decoding to that model. You keep control of the prompt text and lose the built-in task vocabulary.

The practical consequence is what you can change when accuracy is poor. With Promptify, the documented levers are domain, few-shot examples (mentioned in the feature list but not shown in the README examples), and swapping the model. With a schema-binding library, you can rewrite the instruction itself, which is often the fix when a task class's built-in prompt does not match your data. Conversely, if your task is one of the thirteen in the supported table, Promptify saves you from writing and maintaining that prompt at all, and the uniform return types make downstream code shorter.

A second alternative is simply the provider SDK alone with manual Pydantic validation. That is more code, but it makes every failure mode visible, which matters if you need to know exactly when the model did not comply rather than when the parser rescued it.

Licence, maintenance, and what to check before committing

Promptify is released under Apache-2.0, which permits commercial use, modification and redistribution provided the licence and notices are preserved. That is a permissive choice with no copyleft obligation on your own code. This is a description of the licence text, not legal advice; if you redistribute the library or bundle it into a product, have your own counsel review the notice requirements.

The maintenance signal available here is limited. The last push recorded is 2026-03-27, and no releases were retrieved, so there is no tagged version history to reason about upgrade cadence. The README points contributors to contribute.md and directs community questions to a Discord server, which suggests active channels but tells you nothing about release discipline.

Upgrade cost is concentrated in two places. The Pydantic version you pin matters, because every task result is a Pydantic model and major Pydantic releases have historically changed model construction. The LiteLLM version matters for the same reason at the provider layer: a LiteLLM change can alter how a model string resolves or how a provider's structured-output mode behaves, without any change in Promptify's own API. Pinning both, and pinning promptify itself to a PyPI version rather than the git URL, is the cheap insurance. The promptify[eval] extra is a third pin, since the metric implementations come from that dependency set.

What to verify first, concretely: run the NER example from the README against two different model strings, one with native structured outputs and one local via ollama/, and compare the raw responses before parsing. That single comparison tells you which of the two parser paths you will be on in production and how much the fallback completion is doing for you.

Editorial conclusion

Promptify fits teams already committed to Pydantic schemas who want NER, classification, QA or summarization without writing prompt plumbing per provider, and who accept that the model string is the only provider abstraction. It does not fit anyone who needs deterministic, reproducible output without a live model call, or who needs prompt versioning as a first-class artifact, since the README documents neither. Before adopting, run one task against two providers from the LiteLLM list and inspect the raw response path, then check whether promptify[eval] pulls the metric dependencies you actually need.

Official sources

1. Issues
2. License: Apache-2.0
3. Project website
4. promptslab/Promptify on GitHub
5. README