prompt-ops: Meta's CLI for Rewriting System Prompts Against Your Own Examples
An open-source tool for LLM prompt optimization.
At a glance
- What is it?
- prompt-ops is a Python package from meta-llama that takes an existing system prompt, a JSON set of query-response pairs and a YAML config, then emits a rewritten prompt with evaluation metrics. The design is narrow on purpose, and the narrowness is where the real trade-offs sit.
- Who is it for?
- Adopt prompt-ops if you already have a working system prompt, at least roughly 50 query-response pairs, and you are willing to install from source rather than PyPI while the package naming transition is unresolved. Do not adopt it if you have no labelled examples at all, since the documented workflow requires a dataset, or if you need an optimizer that is not tied to the Llama family.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 147 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem prompt-ops targets: prompts written for one model, deployed on another
Most teams do not write prompts from scratch. They copy a system prompt that worked against one model, paste it into a codebase pointed at a different model, and then spend weeks adjusting wording by hand. The README states the package transforms prompts that work well with other LLMs into prompts optimized for Llama models. That is a migration problem, not a greenfield prompt-authoring problem, and the distinction matters for who should care. If you are starting from nothing, this tool has less to offer you, because its input is an existing system prompt file such as use-cases/facility-support-analyzer/facility_prompt_sys.txt. The intended user is an engineer who already has a prompt in production somewhere, a set of examples showing what good answers look like, and a suspicion that the prompt is leaving accuracy on the table. The README frames the pain as manual trial and error. That framing is honest about scope: this is a tool for people whose bottleneck is iteration speed on an existing artifact, not for people deciding what the artifact should be.
Three inputs, one command: the migration pipeline as documented
The architecture diagram in the README is unusually literal, and it is worth reading as a data-flow spec rather than marketing. Three boxes feed a single box labelled prompt-ops migrate: an existing system prompt, a set of query and response pairs, and a YAML configuration. One arrow leaves. There is no feedback loop drawn, no intermediate artifact the user inspects, and no branch for human review. The README describes the workflow in five numbered steps: bring your prompt, prepare a dataset, configure optimization, run one command, receive a model-optimized prompt with performance metrics. The evaluation stage is therefore inside the tool, driven by the dataset you supply and the metrics the config selects. That is the mechanism, and it has a consequence the README does not dwell on. Because the dataset is both the optimization signal and the reported score, the numbers you get back describe fit to your examples, not generalization to traffic you have not sampled. The documentation does not describe a held-out split, so treat the reported metrics as a comparison between candidate prompts on the same data, not as a forecast.
Getting it running: source install, the create command, and the .env key
The README recommends installing from source, and gives the reason directly: the project is transitioning package names on PyPI, and a source install avoids naming conflicts. The documented sequence creates a Python 3.10 conda environment named prompt-ops, clones the repository, and runs pip install -e . from the checkout. The PyPI alternative is listed as pip install llama-prompt-ops, with a note that it may have naming transition issues and is still on version 0.0.7. That version pin is the single most useful line in the install section, because it tells you the published artifact and the repository are not necessarily in step. After install, prompt-ops create my-project scaffolds a directory containing a sample configuration and dataset, and you cd into it. The sample config is referenced as configs/facility-simple.yaml in the repository, which is where you should look for the actual key names, since the README does not enumerate them. The API key step is a single line in a .env file: OPENROUTER_API_KEY=your_key_here. The README states that prompt-ops uses LiteLLM as a unified API client, so the model routing is delegated rather than implemented in the package. That detail matters when you swap providers, because the failure will surface inside LiteLLM, not inside prompt-ops.
Where the tool stops: dataset cost, provider coupling, and the PyPI gap
The first constraint is the dataset. The README sets the floor at as few as 50 examples, and calls the file a JSON collection of query-response pairs. Fifty examples is cheap to assemble if you already log production traffic and can label it, and expensive if you cannot, because the labelling is the work. A team with no labelled data has no path through the documented workflow. The second is provider coupling. Every documented example routes through OpenRouter via a single environment variable, and while LiteLLM supports other backends, the README does not walk through an alternative. If your organization cannot send prompt content and query-response pairs to an external API, this installation path does not fit without work the README does not describe. The third is packaging. A tool that asks you to clone and install in editable mode is a tool whose users are expected to read the source. That is fine for an evaluation, and awkward for a CI pipeline where you want a pinned, reproducible dependency. The README's own warning about the naming transition is the clearest signal that this is not yet a set-and-forget package. None of these are defects in the optimization method. They are the cost of the current distribution.
What PDO adds, and what the label-free claim actually buys you
The README leads with a paper announcement: Prompt Duel Optimizer, described as an efficient label-free prompt optimization method using dueling bandits and Thompson sampling, with reported state-of-the-art results on BIG-bench Hard and MS MARCO. A use case directory, use-cases/web-of-lies-pdo/, is given as a worked example on logical reasoning tasks. The interesting word is label-free. The main workflow described above depends on a query-response dataset, which is a labelling dependency, and PDO is presented as a method that removes it. Those two things sit in tension in the same README, and the documentation does not resolve it: it is not stated whether PDO is a drop-in mode selected through config.yaml, a separate entry point, or a research artifact shipped alongside the main tool. The paper is on arXiv as 2510.13907 if you want to check the method itself. For an adoption decision, the practical reading is that the label-free path exists as a published method and a use case, and the dataset-driven path is the one with a documented end-to-end command. Plan around the second and treat the first as something to verify in the repository before you depend on it.
The alternative worth comparing: DSPy and programmatic prompt compilation
The obvious comparison is DSPy, which also treats prompts as artifacts to be optimized rather than text to be edited. The difference in approach is structural. DSPy asks you to express your task as a program of signatures and modules, then compiles that program against a metric you define. You rewrite your pipeline to fit the framework. prompt-ops asks for the opposite: keep your existing system prompt as a text file, bring examples, and let a migration command rewrite the prompt in place. That is a lower-ceremony path and a narrower one. With DSPy you get a reusable optimization loop you can apply to new tasks and new metrics; with prompt-ops you get a one-shot transformation of a specific prompt, plus whatever the config exposes. If your problem is one prompt that needs to work better on Llama, the prompt-ops shape is a better fit because you do not have to restructure working code. If your problem is that prompt quality varies across a dozen call sites and you want a single optimization discipline, DSPy's model of compiling a program is the more honest abstraction, and prompt-ops will feel like a per-prompt script. Neither is wrong. They optimize different units of work.
Maintenance, licensing and what the repository state tells you
The repository is MIT licensed, which is permissive and places few obligations on how you redistribute or modify the code. That is a statement about the licence text, not legal advice; if you are embedding this in a product, read the LICENSE file and your own counsel's view rather than an article. The last push recorded is 2026-04-21, and no recent releases were retrieved, which is consistent with the README's own note that the PyPI artifact is still on 0.0.7 while the source tree moves ahead. For a maintenance estimate, that combination is the thing to weigh: an active source tree with a lagging published package means your upgrade path is git pull plus pip install -e ., not a version bump in requirements.txt. Budget for reading diffs on the config schema and the CLI surface, because those are the two contracts your project depends on and neither is versioned in a way you can pin from PyPI today. The paper announcement also implies the optimization methods themselves are still being added, which is good for capability and bad for interface stability. Pin a commit hash in your own environment if you need reproducibility, since the editable install from a branch will not give you one by default.
Editorial conclusion
Adopt prompt-ops if you already have a working system prompt, at least roughly 50 query-response pairs, and you are willing to install from source rather than PyPI while the package naming transition is unresolved. Do not adopt it if you have no labelled examples at all, since the documented workflow requires a dataset, or if you need an optimizer that is not tied to the Llama family. Before committing, run prompt-ops create on a throwaway directory and confirm three things: that the generated config.yaml matches the schema in configs/facility-simple.yaml, that your .env actually exports OPENROUTER_API_KEY, and that the metrics the tool reports are the ones your team already uses to judge the prompt.
Community notes