Flaky Test Detector: Scoring JUnit Results and Using a Strands Agent for Root Causes

AICommunity Digest0 replies

Community DigestMember#1

Alicia Marianne Gonçalves (@m4rri4nne) published an article on DEV Community describing a tool she built to detect flaky tests and an optional AI agent that analyzes their failure messages. The original post is available at https://dev.to/m4rri4nne/detecting-flaky-tests-and-finding-their-root-causes-with-strands-agents-and-gemini-4kan and the code lives in the flaky-test-detector repository on GitHub.

The article starts from a familiar problem: AI has made writing automated tests cheaper, so the new challenge is flaky tests, which pass and fail without any change to the code or the test. The author argues that in large projects these tests erode trust in the test suite, cause real failures to be ignored, and increase CI time and cost per pipeline.

How the detector scores tests

The tool parses JUnit XML reports, where each executed test appears as a <testcase>. A FlakinessScorer groups results by test name and then by commit, and marks a test as flaky when the same commit produced both passing and failing outcomes. Skipped tests are ignored, as are results without a commit_sha, since they cannot be compared. Both failed and error count as failures.

The score is the number of flaky commits divided by the number of commits with usable data for that test. A separate FlakinessRanker only considers tests with data from at least five commits (min_commits) and flags a test when its score reaches 0.1 (threshold); both values are configurable. The author illustrates the score with a test that ran on 10 commits and both passed and failed on 3 of them, giving 0.3.

Reports are written in JSON, CSV and Markdown. The JSON format is the one consumed by the agent and includes the generated timestamp, threshold, counts of parsed results and analyzed tests, the flaky rate, and per-test details such as score, commits, flaky commits and distinct error messages.

The Strands agent

The AI layer is built with the Strands Agents SDK and a GeminiModel (the API key is read from GOOGLE_API_KEY in .env), though the author notes other model providers can be swapped in. The agent is given two tools: fetch_json_report, which retrieves reports by timestamp or period and returns them oldest to newest, and export_analysis, which saves the analysis as a Markdown file. Its system prompt instructs it to read the error messages of each flaky test and infer likely root causes such as timeouts, race conditions, network instability or shared state, stating which tests each cause affects and how confident it is.

The agent is invoked from the command line with a period argument, for example python agents/agent.py last_7_days, and a sample analysis report is linked from the repository.

Adoption notes and limitations

The author suggests running the analysis weekly for teams deploying continuously, on every deployment for monthly release cycles, and notes the AI step is optional. Limitations include the requirement that JUnit XML files carry commit_sha and run_id properties, a GitHub Actions-oriented design that may need changes for other pipelines, and a bundled weekly workflow that uploads reports as artifacts.

No replies yet