Hysen Labsself-reflection
Self-Reflection in LLM Agents: Effects on Problem-Solving Performance
Does self-reflection make LLMs better problem solvers?
A study artifact on self-reflection in large language models. Nine models answered multiple-choice questions, first with a baseline agent and then after reflecting on their wrong answers, and the repo ships the abstract, code, and data.
The question being asked
The repository is the study Self-Reflection in LLM Agents: Effects on Problem-Solving Performance. The abstract lays out the design: the study investigated the effects of self-reflection in large language models on problem-solving performance by instructing nine popular LLMs to answer a series of multiple-choice questions, which establishes a performance baseline.
Three steps in the code
The code pipeline is described in three steps. First, solve with baseline, where all questions are answered using the baseline agent. Second, reflect on solution, where the model self-reflects on incorrectly answered problems given the correct answer. Third, save reflections, where reflections are separated by type and redacted. Each step feeds the next.
What the data looks like
The data section covers what the study produces. Details are the low-level details for each answered question in CSV format. Dialogs are the dialog for each answered question in JSON format. Exams contain the multiple-choice questions in JSONL format, and logs are the log files. Each data type has its own format, which keeps the artifact clean.
How the repo is organized
The repository is organized as a research artifact around the abstract, the code, and the data. The design compares a baseline agent against a self-reflecting agent across nine models, and the README presents the materials needed to reproduce the experiment. No results are stated in the README beyond the methodology, so the description stays with the procedure and the file layout.
Community notes