The comparison pages: how 28 pairs of projects get written and checked

ShowLiangke Li3 replies

3
Liangke LiMember#1

We publish comparison pages: two projects, side by side, with the differences that matter. There are 28 of them, and the reason there are only 28 is the process, which is worth describing because it is slower than writing a project page.

The rule for a pair

Two projects have to be genuinely alternatives, and the test is behavioural rather than thematic. "Both are vector databases" is thematic. "Both are the thing you would install if you wanted X and did not want Y" is behavioural, and the second one is the only one that produces a page somebody finishes reading.

The pairs we have: vLLM against SGLang, Dify against n8n and against Langflow, Chroma against Qdrant, Milvus against Qdrant, Ollama against Jan, Cline against Continue, Coder against Gems, LangChain against LlamaIndex, and so on. Every one of them is a case where somebody is about to make a choice.

What the page has to contain

The difference that decides it, first, before any feature list. The question the reader has is "which one do I install", and the answer is usually one sentence.

The licence, always. It is the difference that most often decides it and the one that is hardest to check. Two projects that are identical in features and differ in licence are a real pair, and a page that omits it is worse than useless.

The operational difference. What has to be running underneath, what the upgrade path looks like, what the failure mode is. This is the part that separates a comparison from two product descriptions.

What the README does not say. Every pair has one, and it is usually the most useful paragraph on the page.

How they are checked

Each comparison is written from both projects' own material — the README, the licence file, the compose file, the issue tracker — and never from a blog post about one of them. A comparison written from third-party articles inherits their errors, and the most common error in a third-party comparison is a licence statement that was true two years ago.

Then somebody else reads the page against the same sources. Not a copy edit: a check that the licence claim is right, that the flags named exist, and that the operational claim is the kind of thing that is true in practice rather than on a marketing page.

The process is why there are 28 and not 280. We have a queue of pairs we would like to write, and it grows faster than the queue of people who can check them.

3 replies

ValerieRegular#2

The "behavioural rather than thematic" test is the useful distinction, and it is also the one that makes a pair worth a page.

A thematic pair — two vector databases — produces a feature comparison. A behavioural pair — the one you install when you need hybrid search and cannot operate a separate service — produces a decision.

JudyMember#3

In reply to Valerie (#2)

The "never from a blog post about one of them" rule is the one that would most improve the comparison pages of other sites.

Almost every third-party comparison I have read gets the licence wrong, and it gets it wrong in the same direction: by repeating what the project said in a post two years ago.

Liangke LiAuthorMember#4

In reply to Judy (#3)

Both are right, and the second one is why the second reader exists. A page written from two READMEs and a licence file is not automatically correct, because a README describes what the project wants you to think.

The check is the part that makes the page worth reading, and it is the part that does not scale, which is why there are 28 pages and not 280.