EDSL: running surveys against LLM agents instead of human panels

The problem EDSL is aimed at

Running a survey normally means writing a questionnaire, recruiting respondents, and waiting. Running the same survey against a language model normally means writing prompt strings, parsing whatever comes back, and hoping the output shape stays stable across a few hundred calls. EDSL targets the second workflow and tries to make it look like the first. The README frames the package as a way to "design and run surveys and experiments with many AI agents and large language models at once," and also lists data labeling as a use case. That is the honest scope: it is a tool for generating and structuring model responses, not a survey platform with a sample frame. The audience is computational social scientists, market researchers doing early instrument testing, and engineers who need labeled data in a consistent schema. The README is explicit that agent responses "reflect statistical patterns, not the actual opinions of any demographic group," and recommends using simulated responses for prototyping and pre-testing while validating with real human data when measuring actual attitudes. That caveat is doing a lot of work, and it should be the first thing a new user reads.

Questions, scenarios and agents as separate objects

The design separates three things that prompt-based scripts usually mash together. A Question object holds the wording, the answer options, and the expected answer type. A Scenario is a row of parameter values that get substituted into the question text. An Agent is a set of traits that gets attached to the model call. The README example builds a QuestionMultipleChoice with question_name, question_text and question_options, calls run(), and then selects the answer column. Parameterization works through Jinja-style placeholders: a QuestionLinearScale asks "How much do you enjoy {{ scenario.activity }}?" and a ScenarioList built with from_list("activity", ["coding", "sleeping"]) supplies the values. The q.by(sl).run() call multiplies the question across scenarios, and results.select("activity", "example") returns a table with one row per scenario. Agents work the same way through AgentList and q.by(al).run(), with traits such as persona carried into the output as agent.persona. The mechanism is combinatorial: questions crossed with scenarios crossed with agents crossed with models. That is where the scale claim comes from, and it is also where cost comes from, since every combination is a separate model call.

What the question types buy you

The stated benefit of declared question types is that you get consistent results without writing a JSON schema. That is a real convenience: instead of prompting for structured output and validating it yourself, you pick a type such as QuestionMultipleChoice, QuestionLinearScale or QuestionList, and the package handles the formatting. QuestionLinearScale accepts question_options as a list of numeric values plus an option_labels mapping, so the model sees labels while the stored answer is a number. QuestionList takes max_list_items, which bounds the length of the returned list. The README shows QuestionList returning Python lists such as ['Green', 'Earthy Brown', 'Sunset Orange'] for a botanist persona, which is the kind of output that is annoying to extract reliably from free-form text. The trade-off is that you are limited to the types the library defines. If your instrument needs a ranking, a matrix, or a nested grid, you are dependent on whether that type exists in the docs, and the README does not enumerate the full set. Check the question types page before designing an instrument around this.

Installing it and the account question

The install is a single command: pip install edsl. Python 3.9 through 3.13 is the supported range, and that is stated in the requirements section rather than implied. Beyond the package, the README describes a signup flow: create an account at expectedparrot.com to run surveys on the Expected Parrot server and to access a "universal remote cache" of stored responses used for reproducing results. You then choose between your own API keys for language models and an Expected Parrot key that covers all available models, with key, expense and usage management handled from the account. The starter tutorial and the demo notebooks are the documented next step. This is the part of the design worth pausing on. The remote cache is what makes the replication claim possible: identical questions and inputs can hit stored responses instead of new model calls, which the README describes as results that "can be replicated at no cost." That is a genuine architectural decision, and it also means part of your workflow is tied to an external service. If you need everything to run inside your own infrastructure, the caching story is the constraint to investigate first.

Where the approach breaks down

The failure mode is not technical, it is interpretive. A persona trait such as "detective" does not produce a detective's opinions; it produces text that a model associates with the word. The README says this directly, which is more candor than most projects in this space offer, but the risk remains that a clean results table invites over-reading. A second limitation is cost and latency at scale. The combinatorial design means a study with 50 scenarios, 20 agents and 3 models is 3,000 calls, and nothing in the material suggests batching or sampling shortcuts beyond the cache. Third, the cache only helps on repeat runs of identical inputs; the first run pays full price. Fourth, model behavior changes when providers update weights, so a cached result and a fresh result may not be comparable, and the README does not describe version pinning for models. Finally, if your goal is measuring real attitudes or behaviors, this is the wrong tool by the project's own statement, and no amount of agent trait design fixes that.

How it differs from writing prompt loops yourself

The obvious alternative is a thin script over a model provider's SDK: build a list of prompts, call the API in a loop, write the responses to a DataFrame. That approach is more flexible, has no extra dependency, and lets you use any output format you can parse. It also means you own the prompt templating, the retry logic, the answer parsing, the cost tracking and the reproducibility story. EDSL's value is that those pieces are already objects with names, and that results come back as a dataset with methods for analysis and visualization rather than as a list of strings. The remote cache is the sharpest difference: a hand-rolled loop gives you no way to reproduce a run without paying for it again, unless you build your own caching layer. If your prompts are one-off and your output parsing is trivial, the DSL is overhead. If you are running the same instrument across many scenarios, agents and models, the structure pays for itself in the first study.

Maintenance, licensing and upgrade cost

The package is MIT licensed, which is permissive and places few obligations on how you use or redistribute it; that is a description of the license text, not legal advice, and if you are shipping it inside a commercial product you should read the license yourself. On maintenance, the repository is not archived and the last push recorded is 2026-09-10, so the project is active as of that date. The release list supplied here contains only one entry, a WASM wheel from 2026-03-13, which tells you the packaging surface is being worked on but says nothing about the cadence of the Python package itself. The upgrade cost that matters is not the library version, it is the model version: because results depend on whichever LLM you point at, a provider-side model change can shift your outputs without any change to your code. Keep the model name and the run date alongside your saved results. The remote cache is your main defense here, since it lets you regenerate a prior result set without new calls, provided the inputs match exactly.

Editorial conclusion

Adopt EDSL if you already write Python, need typed question objects rather than raw prompt strings, and treat simulated responses as a prototyping or labeling tool rather than as measurements of real populations. Do not adopt it if your study requires actual human attitudes or behaviors, or if you cannot accept a dependency on an external service for the remote cache and the shared Expected Parrot key. Before committing, verify three things: that the question types you need exist in the docs, that your Python version falls inside the supported 3.9 to 3.13 range, and whether you will run with your own model keys or an Expected Parrot key, since that choice determines both your cost model and how much of your workflow leaves your machine.

Official sources

1. expectedparrot/edsl on GitHub
2. License: MIT
3. Project website
4. README
5. Releases