PyTextRank: A spaCy Pipeline Extension for Graph-Based Phrase Extraction

The problem PyTextRank addresses: ranking phrases inside a spaCy Doc

Extracting the phrases that carry a document's meaning is a recurring task in search indexing, tagging and content analysis. The naive approaches are either statistical (term frequency, which favours single tokens and ignores multi-word units) or embedding-based (which requires a separate model and does not produce a discrete phrase list). PyTextRank takes a third route: it builds a graph over the linguistic units spaCy has already identified and ranks the nodes with a PageRank-style algorithm. The README lists four algorithms in this family (TextRank, PositionRank, Biased TextRank and TopicRank) and three use cases: phrase extraction, low-cost extractive summarization, and inferring concepts from unstructured text into a more structured representation. The intended user is someone who is already running spaCy and wants ranked phrases attached to the Doc they just processed, without standing up a second pipeline. That constraint shapes everything about the library, including its limitations.

How the textgraph is built and what doc._.phrases returns

PyTextRank registers itself as a spaCy pipeline component. The README example calls nlp.add_pipe("textrank") after loading a model, then runs the document through nlp(text). The component reads the tokens, part-of-speech tags and dependency parses that spaCy has already produced, constructs a graph from them, and writes the ranked result back onto the Doc under a custom attribute. The README's loop is the clearest statement of the output shape: for phrase in doc._.phrases, then phrase.text, phrase.rank, phrase.count and phrase.chunks. So each result is an object with a surface string, a numeric rank, an occurrence count, and a list of the underlying chunks that were merged into it. That last field matters: a phrase is not a raw n-gram, it is a grouping of spaCy chunks, which means the tokenizer and the parser determine the boundaries. The ranking itself is graph centrality, so a phrase scores highly when it co-occurs with other highly ranked phrases, not simply when it appears often. This is why the count and the rank are reported separately: a phrase can appear once and still rank near the top if it sits in a dense region of the graph.

Installing PyTextRank and wiring it into a spaCy pipeline

The README gives two installation paths. From PyPI: python3 -m pip install pytextrank followed by python3 -m spacy download en_core_web_sm. Working from the Git repository instead requires python3 -m pip install -r requirements.txt, or conda env create -f environment.yml followed by conda activate pytextrank. The README explicitly notes that unless you are contributing code, you will not need to build the package locally. The usage pattern is three lines of setup: import spacy and import pytextrank, load a model with nlp = spacy.load("en_core_web_sm"), then nlp.add_pipe("textrank"). Note that the import of pytextrank is what makes the "textrank" pipe name resolvable; the component is registered as a side effect of the import. The README points to example notebooks in the examples subdirectory and to the tutorial section of the online documentation for integration patterns with related Python libraries. It also documents a semantic versioning convention worth reading before you pin a version: the major release number of PyTextRank tracks the major release number of the associated spaCy version. That is a maintenance signal, not a compatibility guarantee, and it is the kind of thing you want to check against CHANGELOG.md before an upgrade.

Where PyTextRank is the wrong tool

The library inherits every weakness of the pipeline it plugs into. Phrase boundaries come from spaCy's tokenizer and parser, so domain vocabulary that the model was not trained on will be segmented badly, and no amount of graph ranking repairs a phrase that was split in the wrong place. The README frames the summarization use case as extractive, which means the output is selected source text, never new sentences; if you need abstractive summaries, this is not the component for that. TopicRank is listed among the supported algorithms, but the README does not describe how it is selected or configured at runtime, so treat that as something to verify in the online documentation rather than assume. There is also a scale question the README does not answer: graph construction over a large document means a large graph, and the material supplied here contains no statement about memory use or throughput on long inputs. Finally, the four algorithms are all graph-centrality methods over the same underlying parse. If your phrases are not distinguishable by co-occurrence structure, switching between TextRank and Biased TextRank will not change much.

How PyTextRank differs from YAKE and KeyBERT

The closest alternatives are YAKE and KeyBERT, and they differ from PyTextRank at the level of what they consume. YAKE is a statistical keyword extractor: it scores candidate terms from corpus statistics and does not require a trained language model or a dependency parse, which makes it cheaper to run and easier to apply to languages where no spaCy model is available. KeyBERT derives keyword candidates from document embeddings and ranks them by cosine similarity to the document vector, so it captures semantic relatedness rather than graph centrality, at the cost of loading a sentence-transformer model alongside your existing stack. PyTextRank sits between them in resource terms: it needs a spaCy model with a parser, but nothing beyond that, and it returns ranked phrases as a byproduct of a pipeline you may already be running. The practical difference is what you get back. YAKE gives you scored terms, KeyBERT gives you terms with a similarity score against a document embedding, and PyTextRank gives you phrase objects carrying rank, count and the constituent chunks, attached to the Doc. If your downstream code already iterates over spaCy Docs, that attachment is the whole argument for choosing it.

Maintenance, licensing and the spaCy version coupling

PyTextRank is MIT licensed, and the README states plainly that the licence is succinct and simplifies use in commercial applications. That covers the source code, the logo, the documentation and the examples. The repository is not archived, and the most recent release listed is v3.3.0 from February 2024, following v3.2.5 in August 2023 and v3.2.4 in July 2022. The cadence is irregular rather than dormant, which is typical for a library whose surface area is bounded by an upstream dependency. That dependency is the real maintenance cost. Because the major version tracks the spaCy major version, a spaCy major upgrade is the event that should trigger a PyTextRank review, and the README directs you to CHANGELOG.md for the mapping. The repository also carries mypy and bandit badges and a CI workflow, so the project does run type checking and a security linter on its own code. None of this tells you whether the library will keep pace with a future spaCy release; the README offers no commitment on that point. If you cite the library in research, the README provides a BibTeX entry and a Zenodo DOI, and asks for attribution.

Who should adopt PyTextRank, and what to check first

The fit is narrow and clear. You are processing text with spaCy, you want a ranked phrase list per document, and you would rather not add a second model or a second extraction library to the stack. The README example is short enough to run in a few minutes against your own corpus, and that is the first thing to do: check whether phrase.chunks groups your domain terms the way you expect, because that grouping is the output and it depends entirely on the spaCy model you loaded. The second thing to check is the algorithm selection. The README lists four algorithms but does not document the runtime switch between them, so read the online documentation at derwen.ai/docs/ptr before assuming a default. The third is the spaCy version you are pinned to, since the major versions move together. If any of those three checks comes back wrong for your corpus, the cost of switching to YAKE or KeyBERT later is small, because all three produce a list of scored phrases at the end.

Editorial conclusion

Adopt PyTextRank if you already run spaCy and want ranked phrases on the Doc object without adding a second NLP stack; the MIT licence and the fact that the major version tracks the spaCy major version both reduce integration friction. Do not adopt it if you need abstractive summarization, cross-document topic modelling, or a model that runs without a spaCy pipeline. Before committing, run the README example against your own text and inspect phrase.chunks and phrase.count, since the quality of the output depends on the spaCy model and on how the tokenizer handles your domain vocabulary.

Official sources

1. DerwenAI/pytextrank on GitHub
2. License: MIT
3. Project website
4. README
5. Releases