OpenSSF Scorecard: What the Automated Security Checks Actually Measure

The Gap Between a Dependency You Trust and a Dependency You Have Checked

Most teams decide whether to take on a dependency by reading the README and glancing at the release history. That tells you what the project does. It says almost nothing about whether the maintainers sign their releases, whether the build runs in a hardened CI environment, whether a single person has admin rights, or whether the repository has been archived. Those are the questions that decide how much damage a compromised release would cause, and they are exactly the questions nobody has time to answer by hand for every transitive dependency.

Scorecard was built for that gap. The README states the tool assesses a number of heuristics, called checks, associated with software security and assigns each check a score of 0 to 10. Two audiences are named explicitly: open source maintainers who want to improve their security practices, and open source consumers who need to judge whether their dependencies are safe. If you are a maintainer, the output is a to-do list of specific behaviors. If you are a consumer, the output is a screening signal you can apply across hundreds of repositories without reading any of them.

The framing matters. The project lists as a non-goal being a definitive report or requirement that all projects should follow, and notes that the checks are heuristics with false positives and false negatives. Treat the score as a prompt to look closer, not as a result.

How the Checks Are Produced and Where the Data Lands

Scorecard is written in Go and the module path in go.mod is github.com/ossf/scorecard/v5, so the current code line is v5 even though older documentation and badges still reference v4. The repository is organized around the pieces you would expect from that design: a checks/ directory holding the check implementations, a probes/ directory, a clients/ directory for the platform integrations, a policy/ directory, and a cmd/ directory for the command line entry point. The dependency list shows the shape of the work: go-git for repository access, go-github and a GitLab API client for forge data, actionlint for parsing workflow files, osv-scanner, and in-toto attestation libraries.

Each check runs against a repository and produces a score from 0 to 10. The checks are grouped into a single aggregate score, and the README is direct about what that costs you: aggregate scores in particular tell you nothing about what individual behaviors a repository is or is not doing, and there are multiple ways of arriving at the same score. Two repositories with the same total can have entirely different weaknesses.

V5 introduced structured results as a way around that. The README describes the use case with a concrete example: instead of relying on an aggregate score of X/10, a consumer may want to ensure the repository they depend on is not archived, which is covered by the archived probe. That is a boolean question, and asking it directly avoids the aggregation problem entirely. The OpenSSF states it takes this approach with its own Security Baseline for projects.

There is also a hosted side. Scorecard publishes a weekly scan of the 1 million most critical open source projects, judged by direct dependencies, into the BigQuery public dataset openssf:scorecardcron.scorecard-v2, with the latest results in the view openssf:scorecardcron.scorecard-v2_latest. A web viewer at scorecard.dev shows scores for projects that are regularly scanned.

Building the Scorecard Binary and Scoring One Repository

The README lists installation under the command line interface section, with prerequisites, installation, authentication and basic usage as separate steps. The project ships a Dockerfile whose final stage copies the built binary to /scorecard and sets it as the entrypoint, and the build stage runs make build-scorecard. So the two paths the repository itself supports are the Makefile target and the container image.

Building from source starts with the module dependencies and then the target the Dockerfile calls.

go mod download
make build-scorecard

The Makefile also exposes a help target that prints every documented target, which is the fastest way to see what else is available.

make help

If you would rather not install Go 1.26, the container image built from the repository Dockerfile runs the same binary with /scorecard as the entrypoint.

The README's basic usage section is the reference for how to invoke the binary against a repository, and the authentication section covers the token needed for forge API access. Scores for projects that are regularly scanned are visible in the web viewer at scorecard.dev without running anything locally.

The Aggregate Score Is the Weakest Part of the Output

The most common way to misuse Scorecard is to put the total in a dashboard and alert on it. The README warns against exactly this, and the warning is unusually blunt for project documentation: the aggregate score tells you nothing about which behaviors a repository is or is not doing, and the scores change as new heuristics are added or existing ones are refined. A number that moves because the scoring formula changed is not a measurement of the repository.

There is a second problem with treating checks as findings. They are heuristics, which the README states plainly, and heuristics produce false positives and false negatives. A repository can fail a check for reasons that have nothing to do with its actual risk profile, and it can pass a check while carrying a weakness the check does not model. Nothing in the tool distinguishes these cases for you.

The third limitation is scope. Scorecard assesses repository and forge configuration: workflow definitions, release artifacts, ownership files, and similar signals. It does not read your source code for vulnerabilities. If your question is whether a specific CVE affects a library you use, Scorecard is the wrong tool and a scanner such as the osv-scanner dependency it already pulls in is the right one. Scorecard answers a different question: is this project set up in a way that makes a compromise less likely and easier to detect.

Finally, the check set is opinionated by design. The README lists as a non-goal being a one-size-fits-all solution and admits that what gets included or excluded generates a lot of discussion, and that it is impossible to build a Scorecard that satisfies everyone because different audiences care about different behaviors. If your threat model differs from the OpenSSF's, some checks will be noise to you.

Scorecard Against a Dependency Review Process or a Vulnerability Scanner

The nearest alternative is a manual dependency review: a person opens the repository, reads the contributing guide, checks the release process, and forms a judgement. That produces a richer answer than any check can, because a human can weigh context. It also does not scale past a handful of repositories per week, and it produces a different answer depending on who did the review. Scorecard trades depth for consistency and coverage. Run it across a thousand dependencies and you get comparable output for all of them; the output is shallower than a careful human read, and it is the same shape every time.

The other alternative is a software composition analysis or vulnerability scanner, which answers what known vulnerabilities exist in the code you are pulling in. That is a different question with a different data source. A scanner matches version ranges against advisory databases; Scorecard inspects repository configuration. You can adopt one without the other, and the two produce non-overlapping findings. The repository's own dependency list includes osv-scanner, which is a hint that the maintainers see the two as complementary rather than competing.

Maintenance Cadence, Licensing and What an Upgrade Costs

The repository is not archived, and the last push was on 2026-09-21. Releases are infrequent and deliberate: v5.3.0 on 2025-09-30, v5.4.0 on 2025-11-14, and v5.5.0 on 2026-04-23. That cadence is worth planning around. If you pin a version, expect to sit on it for months at a time, and expect the check set and scoring to shift between minor releases rather than within them.

The go.mod file requires Go 1.26.0, and the Dockerfile builds from golang:1.26.2. That is a recent toolchain requirement, so the source build path needs a current Go on the machine doing the build. The container image sidesteps that constraint because the build happens inside the image.

The project is licensed under Apache-2.0, which permits commercial and internal use and requires that you preserve the license and attribution notices when you redistribute. The published BigQuery dataset is a separate question from the code license, and the README does not spell out terms for the data itself, so check that before building a product on top of the dataset.

For upgrade cost, the practical risk is not the binary. It is anything you built on top of the output. If you parse the CLI output or the structured results and key off specific check names or score thresholds, a minor release that adds or refines a heuristic can change your results without any change on your side. Pinning the version and re-reading docs/checks.md before each bump is the cheaper habit.

Editorial conclusion

Adopt Scorecard if you maintain a repository and want an external, repeatable opinion on supply chain hygiene, or if you consume dependencies and want a screening signal before you spend review time. Do not adopt it as a pass or fail gate, because the maintainers state the checks are heuristics with false positives and false negatives, and the aggregate score hides which behaviors a repository is or is not performing. Before you wire it into anything, run the CLI against one repository you already know well and read the per-check findings rather than the total.

Frequently asked questions

Official sources

1. License: Apache-2.0
2. ossf/scorecard on GitHub
3. Project website
4. README
5. Releases