We write about a lot of projects, and early on we had a rule: read the README, then write. That rule produced confident pages about projects that did not work, because a README is a document about the project written by the project. Here is the checklist we use now.
Before the first command
Does the licence let us do what we are about to do? Read the file, not the badge. grafana/grafana is AGPL-3.0, which for a page about how to run it is fine and for embedding it in a product is a different conversation.
Is there a compose file, and what does it assume? This is the best maintenance signal in the repository and it is almost never discussed. A compose file that references an old major version of a database, or that mounts a directory the current docs no longer mention, tells you the project is running a setup that is not the one they test.
Does the project's own quick start use a version that is still available? Same idea, applied to the base image.
How big is the first download? A project whose quick start pulls four gigabytes before it does anything is a different commitment than one that pulls forty megabytes, and the README rarely says so.
Before the first page goes live
Which part of this is the project's claim and which is ours? Every number we publish is either read from the repository or computed by us, and the page says which. A benchmark we ran is ours, and it is not the same kind of fact as a star count.
What does the project's own documentation not cover? That is usually the most useful paragraph on a project page: the gap between the README and the thing you are trying to do.
Is the thing we recommend still the thing the project recommends? For a project with 60,000 stars, the popular third-party guides are frequently for a version two major releases old. We check the project's own current docs and not the guides.
The part that keeps surprising us
The compose file. A project's compose file is the state of its supported stack, written by the maintainers, updated when they change it — and it is a machine-readable answer to "what does this project actually run against", which no README paragraph gives you.