Giskard v3: A Modular Python Library for Testing Multi-Turn LLM Agents
🐢 Open-Source Evaluation & Testing library for LLM Agents
At a glance
- What is it?
- Giskard v3 is a rewrite of the open-source evaluation library, split into focused packages for checks and scanning. It targets dynamic, multi-turn agent testing with LLM-as-judge evals and a vulnerability scanner, but requires Python 3.12 and brings its own trade-offs.
- Who is it for?
- Adopt Giskard v3 if you build Python-based agentic systems and need a lightweight, scriptable way to run LLM-as-judge checks and automated vulnerability scans without a heavy platform. Do not adopt it if you still rely on v2's tabular or ML model scanning, which remains unsupported in v3.
- Can I use it commercially?
- Yes. Apache-2.0 is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 1 day ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
What Giskard v3 Actually Solves
Giskard v3 is a Python library for testing and evaluating agentic systems, which the README defines as anything from a single LLM call to a multi-step pipeline. The core problem it addresses is that traditional unit tests do not fit non-deterministic outputs: the same input can produce many valid responses. Giskard offers a scenario API where you define interactions and then attach checks, including LLM-as-judge assertions like Groundedness and Conformity. This is aimed at engineers who need regression testing, RAG quality validation, or safety rule enforcement for conversational agents. The v3 rewrite explicitly drops heavy dependencies from v2, positioning itself as modular and async-first. The target user is a developer who wants to write tests in Python code, not configure a SaaS dashboard.
The Modular Architecture and What Each Package Carries
The v3 architecture is a set of focused packages rather than one monolithic library. The README lists giskard-checks as the stable testing layer, giskard-scan as the vulnerability and quality scanner, and three foundational libraries: giskard-core for shared utilities and telemetry, giskard-llm for provider-agnostic routing, and giskard-agents for orchestration. The design intent is that each package carries only the dependencies it needs. For example, installing giskard alone gives you checks plus core, while adding the scan extra pulls in giskard-scan. Provider SDKs are separate extras, such as openai or anthropic. This is a clear departure from v2, which the README says is no longer actively maintained. The trade-off is that you now have to reason about which extras you need, and the foundational packages are pulled in automatically but rarely used directly, which can obscure what is actually installed.
Writing Your First Scenario: The Quickstart Walkthrough
The README gives a concrete quickstart that shows how to run a groundedness check. You define a target function, here get_answer that returns a hardcoded string, then build a Scenario with a name, an interaction that sends inputs to that function, and a check. The Groundedness check takes a context and an LLM judge evaluates whether the answer is grounded in that context. The example uses asyncio.run because scenario.run() is async. To run it, you need a provider extra installed, such as pip install "giskard[openai]", plus the matching API key. The default judge model is openai/gpt-4o-mini. The scenario produces a result object with a print_report() method. This is a minimal but complete pattern: you can replace the hardcoded function with your own agent, and add multiple checks or interactions for multi-turn conversations.
The Vulnerability Scanner: giskard-scan and Its Scope
giskard-scan is the red-teaming layer that the README describes as a successor to v2's Scan. It includes vulnerability_scan for red teaming, prompt injection, jailbreaks, and harmful content, plus quality_scan for RAG evaluation. The scan extra is separate from the base checks library, so you only install it when you need automated adversarial testing. The README stresses that v3's scanner has no dependency on v2, and that only the legacy scan for tabular or ML models remains v2-only. This is an important boundary: if your work involves classical machine learning models rather than LLM agents, giskard-scan will not help you. The quality_scan is positioned as the successor to v2's RAGET test set generation, but the README does not detail what metrics or methods it uses, so you would need to consult the docs to see if it matches your evaluation needs.
Installation and Environment Constraints
Installation is straightforward via pip, but there is a hard requirement: Python 3.12 or newer. The base install is pip install giskard, which brings in giskard-checks and its dependencies. For scanning, you add the scan extra: pip install "giskard[scan]". For LLM judges, you add a provider extra such as openai or anthropic, and you must set the corresponding API key. The README also mentions telemetry: optional aggregated analytics via giskard-core, with a claim that no prompts or outputs are sent. You can opt out by setting DO_NOT_TRACK=1 or GISKARD_TELEMETRY_DISABLED=1 in the environment or a .env file before import. Setting these later stops further sends but does not prevent the creation of ~/.giskard/id. This is a concrete operational detail that matters for privacy-conscious deployments. The Python 3.12 floor is a real constraint for teams still on older LTS versions.
A Genuine Limitation: v2 Legacy and the Missing Tabular Scan
The most obvious limitation is that Giskard v3 is a fresh rewrite, and the README is explicit that v2 remains available but is no longer actively maintained. If you have existing v2 workflows for tabular or ML model scanning, those will not run in v3. The legacy scan for those models stays v2-only, meaning you would have to maintain a separate v2 installation for that use case. Another limitation is the reliance on LLM judges for many checks, which means you need a provider API key and incur costs for each run. The Groundedness example defaults to gpt-4o-mini, but if you do not have that provider configured, the check will fail. The README does not describe offline or local judge options, so this is a potential blocker for air-gapped environments. Finally, the project is young: v3.0.0 was released in August 2026, and giskard-scan is at 1.0.0, so you should expect rough edges and rapid API changes.
Comparison with an Alternative Approach: Direct Evaluation Frameworks
A real alternative to Giskard v3 is building your own evaluation harness using a framework like pytest with direct LLM calls, or using a dedicated LLM evaluation library such as DeepEval. The key difference is in approach: Giskard provides a structured scenario and check abstraction that manages multi-turn interactions and trace-based assertions, whereas a pytest-based approach requires you to write your own assertion logic and manage conversation state manually. For example, with Giskard you define a Scenario, call .interact() with your agent function, and attach a Groundedness check; the library handles the judge invocation and report generation. With a raw framework, you would write a test function that sends prompts, receives responses, and then calls an LLM judge with your own prompt template. Giskard's advantage is the built-in check library and the scan tooling; the alternative's advantage is full control and no extra dependency layer. The choice depends on whether you want a higher-level abstraction or maximum flexibility.
Maintenance, Upgrades, and License Considerations
The project is licensed under Apache-2.0, which permits commercial use and modification with attribution, but this is not legal advice. Maintenance is active: the last push was September 2026, and v3.0.0 was released in late August 2026. However, the README warns that v2 is no longer actively maintained, so any v2-based code you have will not receive fixes. Upgrading from v2 to v3 is not a drop-in replacement; the architecture has changed from a monolithic library to modular packages, and the scan functionality has moved to giskard-scan. You should budget time to migrate your test suites. The telemetry system is a minor maintenance consideration: you need to remember to set the opt-out environment variables if you do not want to create the ~/.giskard/id file. The modular packaging means you will need to track multiple package versions, but the README suggests the foundational libraries are pulled in automatically, so your pip dependency tree will be larger than before.
Editorial conclusion
Adopt Giskard v3 if you build Python-based agentic systems and need a lightweight, scriptable way to run LLM-as-judge checks and automated vulnerability scans without a heavy platform. Do not adopt it if you still rely on v2's tabular or ML model scanning, which remains unsupported in v3. Before committing, verify that your provider SDK is covered (e.g., openai, anthropic) and that your Python environment is 3.12+, since older versions will not install. Also check the telemetry opt-out mechanism if your organization restricts data collection, and review the giskard-scan package's current maturity, given its recent 1.0.0 release.
Community notes