markdown-crawler: recursive site-to-markdown conversion for RAG corpora
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
At a glance
- What is it?
- A multithreaded Python crawler that writes one markdown file per page, built for LLM document parsing. It is a small, opinionated tool: easy to run, but you have to check what its CSS selectors and path filters actually keep.
- Who is it for?
- Adopt markdown-crawler if you need a flat directory of markdown files from a documentation site or wiki and you are willing to verify the output by hand. Do not adopt it if you need JavaScript-rendered pages, crawl politeness controls, or a single consolidated corpus file.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 81 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on September 15, 2026, and from our analysis. They are not legal advice.
DEEP OPEN-SOURCE ANALYSIS
The problem it solves: turning a documentation site into a chunkable corpus
Retrieval augmented generation pipelines want text that is already structured. Raw HTML carries navigation, cookie banners, sidebars and inline styling that survive naive tag stripping and pollute embeddings. The README states the project was primarily created for large language model document parsing, to simplify chunking and processing of large documents for RAG use cases, and argues that markdown is human readable while keeping a small footprint and preserving document structure.
The intended user is an engineer assembling a corpus, not a general web scraper operator. The README lists RAG as the primary use case, LLM fine-tuning corpora as a second, agent knowledge reconstruction (it names autogen as an example), and online RAG learning where you scrape and index search results. All four assume you want files on disk, one per page, that a later chunker can split by header, paragraph or sentence. If your target is a single concatenated blob, this tool does not produce that.
How the crawl actually works: threads, BeautifulSoup, markdownify
The pipeline is short and the README names every stage. requests fetches a page. BeautifulSoup parses the HTML. A CSS selector passed as target_content narrows the parse to a subtree, and the README notes you can supply multiple selectors and the results will be concatenated. markdownify converts the selected HTML to markdown, with the heading style controlled by heading_style. The result is written to a file under base_dir.
Link discovery drives recursion. The crawler collects anchors from the page, filters them, and queues them for the next level until max_depth is reached. Two filters decide what is in scope: is_domain_match, which the README describes as restricting crawling to pages on the same domain as the base URL, and is_base_path_match, which when false includes all URLs on the domain even if they do not begin with the base URL. valid_paths takes an array of relative paths and limits crawling to those plus the base path. exclude_paths removes paths you name.
num_threads sets the number of parallel workers. The README itself hedges with the phrase parallel(ish) threads, which is a fair description of a queue-based worker pool: ordering of writes is not guaranteed, and the depth counter is what bounds the crawl, not the thread count. Resume support is listed as a feature, so a run that stops can be continued rather than restarted, though the README does not document the exact state file or directory convention used to detect what was already fetched.
Running it: the CLI flags and the library call
Installation is a single pip command. The README gives `pip install markdown-crawler`, then a CLI invocation:
`markdown-crawler -t 5 -d 3 -b ./markdown https://en.wikipedia.org/wiki/Morty_Smith`
Here -t is num_threads, -d is max_depth and -b is base_dir. Running from a checkout is the same after `pip install .` in the repository root.
The full argument list from the README is `--max-depth`, `--num-threads`, `--base-dir`, `--debug`, `--target-content`, `--target-links`, `--valid-paths`, `--exclude-paths`, `--domain-match`, `--base-path-match`, `--heading-style` and `--links`, with base_url as the positional argument. Note the CLI long names differ from the library keyword names: the CLI uses --base-dir and --domain-match, while the Python API uses base_path and is_domain_match. Copying a CLI flag into a function call will not work.
The library form is three lines:
`from markdown_crawler import md_crawl` `url = 'https://en.wikipedia.org/wiki/Morty_Smith'` `md_crawl(url, max_depth=3, num_threads=5, base_path='markdown')`
The README's example configuration sets max_depth to 3, num_threads to 5, base_dir to markdown, valid_paths to an array of relative paths, target_content to div#content, is_domain_match to False, is_base_path_match to False and is_debug to True. Read that combination carefully before reusing it: with is_domain_match False and is_base_path_match False, the crawler will follow links anywhere on the domain, so the path filters are the only thing keeping the crawl bounded.
The selector is the whole game, and the defaults are not safe
Everything downstream of the fetch depends on target_content. If you leave it unset, markdownify converts whatever BeautifulSoup was handed, and on a normal documentation site that includes the sidebar, the footer and the previous/next navigation. Those strings end up in your corpus and in your embeddings. The README's own example pins div#content, which is the right instinct: pick the container that holds the article body and nothing else.
The v0.0.9 release notes show how brittle that selector handling was. Two of the six listed fixes concern it: issue #10, described as target_content CSS selector handling where null href links are now skipped, and issue #11, an UnboundLocalError in get_target_content. Both are the kind of defect you hit when a selector matches nothing on some pages and the code path assumes a match. The current release is dated 2026-06-26 and the previous one, 0.0.8, is from 2023-10-30, so the fixes are recent relative to a long quiet period. That history is worth knowing before you point it at a site whose markup varies page to page.
What it does not do: JavaScript, politeness, and single-file output
The crawler fetches HTML with requests and parses it with BeautifulSoup. Neither executes JavaScript. The v0.0.9 notes list a browser-like User-Agent header added on all requests to bypass JavaScript checks, which addresses servers that reject the default requests agent. It does not address pages whose content is assembled client-side. For a single-page application or a site that renders its article body after hydration, the fetched HTML will be a shell and the markdown output will be thin or empty. That is a structural limit of the fetch layer, not a bug.
The README documents no request delay, no rate limit, no robots.txt handling and no retry policy. num_threads is the only lever on request pressure, and it is a blunt one: five workers against a small origin is a different proposition from five workers against a large one. If you need crawl-delay semantics or robots compliance, you are implementing them yourself around this tool, and the README gives no hook for that.
Output is also one file per page by design. There is no documented merge step, no manifest of what was written, and no documented schema for mapping a markdown file back to its source URL. You get a directory tree, which the README illustrates with a screenshot of the markdown directory. Reconstructing provenance for citation is your problem.
Where it sits against a general-purpose crawler framework
The obvious comparison is Scrapy, which is also Python and also fetches and parses HTML, but the two sit at different levels. Scrapy is a framework: you write a spider class, define item pipelines and exporters, and the HTML-to-text step is something you add. It ships scheduling, concurrency control, retry middleware, AutoThrottle and robots.txt handling as configuration. markdown-crawler is an application: one command, a fixed pipeline of requests to BeautifulSoup to markdownify to disk, and filters exposed as flags.
The trade is real in both directions. Choosing markdown-crawler means you get markdown files in one command and no spider code to maintain, and you accept that the conversion step, the output layout and the request policy are not yours to change without editing the package. Choosing Scrapy means you write the conversion yourself, typically wiring markdownify or a similar library into a pipeline, and in exchange you get throttling, retries and robots handling that markdown-crawler does not document. If your crawl is a one-off corpus build against a cooperative origin, the application is the shorter path. If it is a scheduled job against someone else's server, the framework is the responsible choice.
Maintenance, release cadence and the MIT licence
The release history is uneven. Version 0.0.4 landed 2023-10-24, 0.0.8 on 2023-10-30, and then nothing until 0.0.9 on 2026-06-26. That long gap matters for planning: for roughly two and a half years the package sat at 0.0.8 while issues accumulated, and the current release is the one that cleared six of them. The 0.0.x version numbers are honest about maturity. Treat upgrades as something to test rather than something to assume is safe, and pin the version in your requirements file.
Dependency surface is small: Python 3.x, BeautifulSoup4, requests and markdownify, per the Requirements section. The v0.0.9 notes mention dependencies added to pyproject.toml for uvx support, so the package can be invoked without a permanent install. The release notes also mention a test suite of 60 tests at 95% coverage. That is the project's own figure from its release notes, not an independent measurement.
The licence is MIT, Copyright (c) 2023 Paul Pierre. The grant permits use, copying, modification, merging, publishing, distribution, sublicensing and sale, with the condition that the copyright notice and permission notice are included in all copies or substantial portions. The software is provided as is, with no warranty. Two practical consequences for a corpus pipeline: keep the licence text with any vendored copy, and remember that MIT covers the crawler, not the pages you crawl. Copyright and terms of use for the target site are a separate question that this licence says nothing about.
Editorial conclusion
Adopt markdown-crawler if you need a flat directory of markdown files from a documentation site or wiki and you are willing to verify the output by hand. Do not adopt it if you need JavaScript-rendered pages, crawl politeness controls, or a single consolidated corpus file. Before trusting a run, crawl a small subtree with --max-depth 1 and --debug, then read three generated files and confirm that target_content kept the article body rather than the navigation, and that exclude_paths dropped the sections you did not want.
Community notes