criticality_score: Calculating Open Source Project Importance

What criticality_score measures for open source projects

criticality_score addresses a prioritization problem that ordinary popularity metrics do not fully answer. A repository can have many stars but limited operational dependence, or it can have modest public popularity while serving as a dependency for thousands of other projects. The project defines criticality as influence and importance, represented by a number between 0 and 1. A score of 0 is the least critical value described by the project, while 1 is the most critical value.

The intended users are open source maintainers, security teams, and organizations that need a list of projects for proactive security work. The README names three goals: generate a criticality score for every open source project, create a list of projects on which the open source community depends, and use that list to improve the security posture of those projects. This is not a vulnerability scanner. It does not claim that a high-scoring repository contains weaknesses, nor does it rank vulnerability severity.

The signals include project age, time since the last update, contributors, organizations, commit frequency, releases, issue activity, comments, and dependency evidence. These signals reflect different aspects of importance, but each has a weakness. A dormant project may remain highly depended upon. A busy project may be active without being widely used. Issue counts depend on the behavior of maintainers and users. The README explicitly warns that there will be exceptions to the individual reasoning rules, so the score should be treated as a prioritization signal rather than a final verdict.

The project is maintained by the OpenSSF Vulnerability Disclosures WG. The repository is not archived, and its last push was on 2026-08-27. The README identifies the project as beta, which is relevant when comparing it with mature scoring systems or using its output for high-impact automated decisions.

The Rob Pike scoring model and its weighted signals

The score is based on an algorithm by Rob Pike. The README provides default weights and maximum thresholds, allowing each signal to influence the final normalized value. The largest positive weights shown in the table are assigned to contributor_count and dependents_count, both with a weight of 2. The negative weight for updated_since is -1, while most other activity signals use positive weights ranging from 0.5 to 1.

Contributor_count counts contributors with commits. Its stated purpose is to indicate project importance through the number of people involved. org_count measures the number of distinct organizations to which contributors belong, which is intended to capture cross-organization dependence. These are useful distinctions, but contributor counts can be affected by bots, migrations, affiliations recorded in GitHub profiles, and changes in repository history. The README does not document normalization rules for handling every edge case, so those details should be examined before comparing organizations with very different community structures.

dependents_count is based on project mentions in commit messages. The README calls this a cross-language approximation and notes that it can work for languages such as C and C++ that do not always have package dependency graphs. This is the most interesting engineering choice in the model because it broadens coverage, but it is also the easiest signal to misinterpret. A commit message mention can indicate a real dependency, a tool invocation, an upgrade, or an unrelated reference. Conversely, a dependency can be present without a detectable commit-message mention. The project describes the method as hack-ish, which is a fair warning against treating it as a complete dependency graph.

The remaining signals measure activity within time windows. created_since and updated_since use months, with a maximum threshold of 120. commit_frequency covers the last year, recent_releases_count covers the last year, and closed_issues_count, updated_issues_count, and comment_frequency cover the last 90 days. The configuration can be replaced through the -scoring-config parameter, and the README points to configuration files under config/scorer. That flexibility is useful for research, but changing weights changes the meaning of the resulting score. Scores produced under different configurations are not automatically comparable.

Repository data flow through the Go command

The main command accepts a repository URL and prints repository metadata, collected signals, and a default score. For a GitHub repository, it requires a GitHub access token in the environment. The README contains two authentication examples with different variable names. Its main usage example uses GITHUB_TOKEN, while the Authentication section specifies GITHUB_AUTH_TOKEN. The latter is the documented authentication variable in the authentication instructions. This inconsistency should be checked against the current command behavior before automation relies on it.

The command also offers Google Cloud authentication. The usage example includes an optional gcloud authentication command and notes that it can be skipped for dependency development. This suggests that the project can combine hosted data sources with repository analysis. The README does not fully document the internal package boundaries, but the go.mod file shows dependencies for Google BigQuery, Google Cloud Storage, GitHub, deps.dev, and Scorecard. Those dependencies are consistent with a workflow that collects signals from hosted datasets and repository APIs before calculating a result.

The output shown in the README contains fields such as repository name, URL, language, license, contributor count, organization count, commit frequency, issue signals, dependents count, and default_score. The output is therefore not merely a final number. It provides an audit trail for the signals used in the calculation, which is important when a score influences a security queue. The command supports text, json, and csv output through the -format flag. The repository also contains additional commands, including enumerate_github, collect_signals, and scorer, indicating that collection, large-scale processing, and recalculation are separate workflows.

Installing criticality_score and running a first repository query

The README presents installation with the Go toolchain. Install the versioned command module, then export the authentication variable documented in the Authentication section. The command needs a GitHub token for GitHub repositories. The first query below is copied from the README and includes the optional Google Cloud authentication step.

go install github.com/ossf/criticality_score/v2/cmd/criticality_score@latest

After installation, set the access token using the variable named by the Authentication section. On POSIX systems, the documented command is:

export GITHUB_AUTH_TOKEN=<your access token>

The usage section also shows this Google Cloud login command, which it marks as optional:

gcloud auth login --update-adc

Run the command against a GitHub repository URL. The following is the exact usage form shown in the README:

criticality_score -gcp-project-id=[your projectID] https://github.com/kubernetes/kubernetes

The reader should see repository details, collected signal fields, and a `default_score` line. For output intended for scripts or analysis, select one of the documented formats with `-format`. The README names `text`, `json`, and `csv`, but does not give a complete format-selection command or schema, so verify the exact CLI behavior before building a parser around it.

Data availability and the Google Cloud shutdown boundary

The README contains an important operational warning: all Google Cloud infrastructure and data hosted there will no longer be available after 2026-08-29. It specifically names GCS and BigQuery data. The latest available data is placed in the repository's `data` directory. This changes the deployment and upgrade decision for anyone who planned to rely on the hosted datasets.

The repository still contains data-related directories and code, including `data/`, `infra/`, and commands for collecting signals. That layout does not establish that every feature remains usable after the hosted infrastructure disappears. The README points readers to issue #833 for more information, but the issue text is not included here. It would be unsafe to assume that a self-hosted replacement is complete or that local data reproduces every hosted result.

There is also a date inconsistency within the usage material. The main example refers to GITHUB_TOKEN, while the Authentication section refers to GITHUB_AUTH_TOKEN. The README does not explain whether the main example is stale or whether both variables are accepted. For a reproducible first run, test the documented Authentication variable and inspect the resulting authentication error or output. This is a small example of why setup instructions should be verified against the current CLI rather than copied blindly.

The Apache-2.0 license permits use and modification under its terms, subject to the license and notice requirements. This is an operational note, not legal advice. Review the repository's LICENSE file and preserve required notices when redistributing code or embedding it in a product.

Where criticality_score is the wrong tool

criticality_score is the wrong tool when the question is whether a specific version has a known vulnerability. Its inputs describe project influence and activity, not exploitable defects. It also does not replace a software bill of materials, package dependency resolver, or code review. A high score can identify a project worth examining, but it does not establish that the project is unsafe.

It is also a poor sole basis for ranking projects inside a small application. A dependency may be operationally important to one company without having a large public dependency signal. The default model includes thresholds such as 500,000 for dependents_count and 5,000 for several issue and contributor signals, so scores may compress or behave differently at small scale. The README does not document a confidence interval, uncertainty measure, or validation procedure for individual repositories.

The signal collection introduces another failure mode. GitHub API rate limits, incomplete repository history, renamed projects, archived repositories, missing release metadata, and changes in contributor affiliations can all affect inputs. The README explains why a token is needed to avoid unauthenticated rate limits, but it does not claim that token use eliminates collection gaps. Any automation should therefore retain the raw output and record the date and configuration used for each calculation.

A real alternative is to combine dependency graph data from a package manager or SBOM tool with this score as a separate importance layer. A package-aware approach can identify direct and transitive dependencies more precisely for ecosystems that publish machine-readable metadata. criticality_score's difference is that it combines repository activity with a cross-language mention heuristic, so it can cover projects without a conventional package dependency graph. The cost of that broader coverage is reduced precision.

Criticality score maintenance and upgrade considerations

The repository is not archived, and the last push was on 2026-08-27. That indicates recent repository activity, but it does not guarantee that every hosted feature remains available. The README's notice about the Google Cloud shutdown on 2026-08-29 means that maintenance status and infrastructure status must be considered separately. Recent source changes do not by themselves preserve access to hosted GCS and BigQuery data.

The project uses Go 1.24.2 in go.mod and has a substantial dependency set, including Google Cloud libraries, GitHub client code, Scorecard, and Google API packages. Upgrades can therefore involve both the application and its external services. The repository's Makefile defines test and lint targets, including `go test -race -covermode=atomic -coverprofile=unit-coverage.out './...'` and a Scorecard version validation script. Those targets indicate a conventional Go validation workflow, but the README does not document a supported upgrade procedure or compatibility policy for historical scores.

Users who need stable comparisons should pin the binary or module version, retain the scoring configuration, and keep the raw signal output. A score recalculated after a change in weighting, data availability, or collection behavior may not be comparable with an older score. The README explicitly allows custom configurations, so record that choice alongside each result.

Editorial conclusion

Adopt criticality_score when you need a repeatable way to compare project importance or prioritize security attention across open source dependencies. Do not treat its output as a complete risk assessment or replace dependency inspection with it. First verify authentication, the scoring configuration, the available repository data, and the implications of any Google Cloud services before using the score in a decision process.

Frequently asked questions

Official sources

1. Issues
2. License: Apache-2.0
3. ossf/criticality_score on GitHub
4. README
5. Releases