ai-scanner

What the scanner actually tests

Scanner is a web application for AI model security assessment. It is built with Ruby on Rails and sits on top of NVIDIA garak, a well known LLM vulnerability scanner. The tool helps organizations find weaknesses in their AI systems before those systems reach production, in the same spirit as penetration testing for ordinary software. Rather than shipping only simulated or toy attacks, Scanner bundles real, 0DIN-disclosed jailbreaks as ready to run probes. The README highlights six of these, and each one ships with the original attack prompt plus 34 retargetable variants, with a link to its public threat report. Those reports describe affected models, taxonomy, reproducibility scores, and suggested mitigations. On top of the featured set, the project lists 179 community probes that span 35 vulnerability families. These families are aligned with the OWASP LLM Top 10, so a team can measure coverage against a recognized standard. The probes target both API based LLMs and browser based chat interfaces, which means a single deployment can exercise models served over an API as well as products that only expose a web chat. This breadth matters because the failure modes of the two surfaces differ, and a scanner that covers both gives a more complete picture of where a model can be pushed off track.

Running scans and reading results

Scanner supports both scheduled and on demand scans, and each scan can be configured with its own recurrence. When a scan runs, the tool tracks an Attack Success Rate, or ASR, and keeps history so you can watch that rate move across runs. A Live Activity Stream lets you watch queued and running scans in the web interface, with database backed execution log tails and the final report logs shown inline. Results can be exported as a PDF, and the export supports drill down to individual probes and even to single attempts, which helps when you need to explain a failure to a model owner or a compliance reviewer. For teams that already run a security operations pipeline, Scanner forwards results to a SIEM. The README names Splunk and Rsyslog as supported destinations, so findings land where the rest of the security team already works. On the operations side, the tool is multi tenant, which means several organizations can share one deployment while keeping their data separate, and the data is encrypted at rest. The README also states there are no artificial limits in the open source build. Every feature is available, and there is no cap on the number of scans or users. That removes a common friction point where a free tier stops at a small number of tests and forces a purchase just to keep evaluating.

Installing and getting started

The fastest way to stand up Scanner is a one line install script that the README publishes. Running curl with the project's install.sh and piping it to bash pulls the needed files and sets things up. If you prefer to see every step, the manual path uses a docker compose file and an environment template. You download docker-compose.yml and .env.example from the main branch, copy the example into a local .env, and then edit a few values. The README tells you to set SECRET_KEY_BASE with a freshly generated value from openssl rand, along with POSTGRES_PASSWORD and ADMIN_INITIAL_PASSWORD. After that, docker compose up brings the stack online. You then open the local instance in a browser and log in with admin@example.com and the password you placed in .env. The docs urge you to change that initial password right away. From there the Quick Start guide walks through port configuration, a first scan, and troubleshooting. A First Scan guide explains how to run against the built in Mock LLM so you can learn the workflow without pointing real traffic at a production model. The documentation set also covers core concepts such as targets, scanning, reports, and probes, and it points to a landing page for community and enterprise options.

Editorial conclusion

The project is published under the Apache-2.0 license and is written primarily in Ruby.