# Self-Reflection in LLM Agents: A Reproducible Study of Its Effect on Problem-Solving

> This repository contains the code and data for a study that tests whether eight types of self-reflection improve the accuracy of nine LLMs on multiple-choice questions. The results show a significant improvement, but the setup has clear limitations for real-world use.

**matthewrenze/self-reflection** — Self-Reflection in LLM Agents: Effects on Problem-Solving Performance

- Repository: https://github.com/matthewrenze/self-reflection
- Stars: 104 · Forks: 11
- Language: Python
- License: BSD-2-Clause
- Published: 2026-08-17 · Updated: 2026-08-18 · Language: en
- Canonical page: https://hysenlabs.com/projects/matthewrenze-self-reflection

## What This Study Actually Tests

The repository is not a library or a tool. It is a complete research artifact for a specific experiment: measuring whether self-reflection improves the accuracy of large language models on multiple-choice question answering. The study uses nine popular LLMs as baselines, then introduces eight types of self-reflecting agents. Each agent reflects on its incorrect answers, given the correct answer, and produces guidance for itself. It then re-answers the same questions. The README states that the results show a significant improvement in problem-solving performance through self-reflection. This is a narrow, controlled setup. It does not address open-ended tasks, multi-step reasoning, or real-world decision-making. The target audience is researchers replicating the study or exploring similar hypotheses, not engineers building applications.

## The Pipeline: From Baseline to Statistical Test

The code is organized as a numbered sequence of scripts, each with a clear stage. The first script, 1_solve_with_baseline.py, answers all questions using the baseline agent. The second, 2_reflect_on_solutions.py, takes the incorrectly answered problems and has the agent self-reflect, given the correct answer. This is a critical detail: the reflection is not open-ended; it is guided by the ground truth. The third script, 3_save_reflections.py, separates reflections by type, redacts answers, and saves the text. Redaction is important because the reflection must not leak the correct answer when the agent re-answers. The fourth script, 4_solve_with_reflections.py, re-answers the incorrectly answered questions using the saved reflections. Then three plotting scripts generate accuracy visuals, and two analysis scripts run statistical tests. Script 8 performs the McNemar test, which is a standard method for comparing paired proportions. Script 9 analyzes error keywords in the reflections. This pipeline is linear and reproducible, but it requires manual coordination between steps. There is no single entry point or configuration file.

## Getting It Running: Commands and Data Layout

The README does not list installation commands or dependencies. It only provides the script names and the data directory structure. To run the pipeline, you would clone the repository and execute the scripts in order, presumably with Python. The data directory contains subfolders: details, dialogs, exams, logs, plots, reflections, results, and tables. The exams are in JSONL format, details and results are CSV, dialogs are JSON, and reflections are plain text. This layout suggests that each script reads from one directory and writes to another. For example, script 1 likely reads exams and writes results. Without a requirements.txt or environment.yml, you must infer the required packages from the imports in the scripts. This is a barrier for quick adoption. The scripts are numbered, which implies a strict execution order, but there is no orchestration script to run them all at once.

## The Eight Types of Self-Reflection: A Key Comparison

The study does not just test one form of self-reflection. It compares eight types. The README does not enumerate them, but the scripts and data suggest that each type is a variant of the reflection prompt or the guidance generated. The analysis scripts, such as 9_analyze_keywords.py, look at the error keywords produced by the reflections, which implies that the types differ in how they frame the mistake. This comparison is the core contribution: it isolates which aspect of self-reflection drives the improvement. However, the repository does not provide a summary of the findings beyond the abstract. To know which type performed best, you would need to read the research paper linked in the README. This is a limitation for someone who wants a quick answer without reading the paper.

## A Major Limitation: The Reflection Uses the Correct Answer

The most significant constraint is that the reflection step is given the correct answer. Script 2 is described as 'self-reflects on incorrectly answered problems given the correct answer.' This means the agent is not reflecting in a vacuum; it is told what the right answer is. In a real-world scenario, you do not have the ground truth. This makes the study's results an upper bound on what self-reflection can achieve when the model is given feedback. It is not a realistic test of autonomous self-correction. The README also notes that script 3 redacts answers before saving reflections, which ensures the re-answer step does not see the correct answer, but the reflection itself has already been influenced by it. This is a fundamental design choice that limits the applicability of the findings to settings where external feedback is available, such as grading or evaluation.

## Alternative: Self-Consistency or Verifier-Based Approaches

A common alternative to self-reflection is self-consistency, where the model samples multiple reasoning paths and takes a majority vote. That approach does not require a correct answer or a reflection step; it relies on the model's own diversity. Another alternative is using a verifier model to score candidate answers, which is a different mechanism: instead of the model reflecting on its own mistake, a separate model judges the output. The key difference is that self-reflection here is introspective and guided by the correct answer, whereas self-consistency is purely generative and verifiers are external. This repository does not implement these alternatives, but the contrast is useful for understanding what the study does and does not show. If you need a method that works without ground truth, you would look at those approaches instead.

## Data and Reproducibility: What You Get

The repository includes the full data from the experiment. The data directory has low-level details for each question, the dialogs, the exams, the logs, the plots, the reflections, the results, and the tables. This is a strong point for reproducibility. You can inspect the exact prompts and responses in the dialogs and logs. The plots are available as PDFs, and the results are in CSV format for easy analysis. However, the README does not specify the exact models used or the versions. The abstract mentions 'nine popular LLMs' but does not name them. To know which models, you would need to read the paper or examine the data. This is a gap for someone who wants to replicate the study on the same models. The license is BSD-2-Clause, which is permissive and allows reuse with attribution, but it does not grant any trademark rights or imply endorsement.

## Maintenance and Upgrade Costs

The repository was last pushed at an unknown time, but it is not archived, which suggests it is still maintained. There are no recent releases, so you should expect to use the code as-is. The scripts are plain Python files, so upgrading to a newer LLM API would require editing the code. The README does not mention any dependency management, so you are on your own for installing packages. The cost of maintaining this as a reproducible study is moderate: the data is static, but the code may break if the underlying LLM APIs change. For a one-off experiment, this is fine. For ongoing use, you would need to update the model calls and possibly the data formats. The license is permissive, so you can fork and modify it without legal friction, but you must retain the copyright notice.

## Conclusion

Adopt this repository if you want a reproducible baseline for studying self-reflection in LLM agents on multiple-choice exams. Do not use it as a general-purpose framework for production systems; it is a research artifact with a fixed pipeline and no packaging. Before using it, verify that your Python environment matches the dependencies (not listed in the README), and inspect the data formats in data/exams and data/results to ensure they fit your needs. If you need a production-grade agent loop, look elsewhere; this is for controlled experiments.

## FAQ

### What does the self-reflection repository study?

It investigates the effect of self-reflection in large language models on problem-solving performance. Nine popular LLMs answer multiple-choice questions for a baseline, eight types of self-reflecting agents review their wrong answers, and each agent re-answers the same questions using that guidance.

### Which scripts run the self-reflection experiment in what order?

source/1_solve_with_baseline.py answers every question, source/2_reflect_on_solutions.py reflects on the wrong ones, source/3_save_reflections.py separates reflections by type and redacts the answers, and source/4_solve_with_reflections.py re-answers using them. Scripts 5 to 9 plot accuracy and run the analysis.

### Why does the self-reflection code redact answers from the reflections?

The saved reflection text is reused as input when the agent re-answers the same questions, so the answers are stripped out to keep that second attempt a test of the guidance rather than a lookup. The redacted text is stored as plain text files under data/reflections/.

### What statistical test does the self-reflection analysis use?

source/8_analyze_details.py performs the McNemar test and creates a table of the results, stored as CSV in data/tables/. The same model answers the same question before and after reflection, so the outcomes are paired.

### What licence and data formats does the self-reflection repository use?

The repository is under the BSD 2-clause licence. Details and results are CSV, dialogs are JSON, exams containing multiple-choice questions are JSONL, logs and reflections are plain text, and the visualisations in data/plots/ are PDF.

## Sources

- [Official README](https://github.com/matthewrenze/self-reflection#readme)
- [Project repository](https://github.com/matthewrenze/self-reflection)
- [README](https://github.com/matthewrenze/self-reflection/blob/main/README.md)
- [Releases](https://github.com/matthewrenze/self-reflection/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/matthewrenze-self-reflection