PullMD: A Self-Hosted Anything-to-Markdown Service for Humans and Agents

The gap PullMD fills between a raw fetch and a usable document

Fetching a URL is trivial. Getting Markdown out of it that an agent can consume without burning context on navigation, ads and boilerplate is the actual work. PullMD is built around that second step. The README describes it as a self-hosted URL-to-Markdown service for humans and AI agents, and the v3 line extends the input surface from web pages to PDF, Office, EPUB, images, audio and YouTube.

The audience is narrow and identifiable. Someone running a Claude Code workflow, an MCP client, or any agent pipeline that needs page content as text. Someone who wants the conversion to happen on their own hardware rather than through a third-party API. Someone who has hit a JavaScript-heavy page that a plain HTTP fetch returns as an empty shell.

The project ships four interfaces over the same conversion core: a PWA frontend, a REST API at GET /api?url=, an MCP server at POST /mcp using Streamable-HTTP transport in a stateless configuration, and a Claude Code skill distributed as a downloadable zip. That breadth is the pitch. One service, several entry points, one output format.

How the extraction pipeline decides what to run

The README lays out a cascade. Cloudflare's native Markdown is used when available. Static HTML goes through Mozilla Readability plus Trafilatura. JavaScript-heavy pages fall back to headless Chromium via a Playwright sidecar before extraction. Reddit and Hacker News threads are auto-detected and handled by purpose-built converters that include full comment trees.

That ordering matters for cost. The cheap paths run first, and the expensive renderer is a last resort. The v3.10.1 release note says the renderer stopped announcing itself as HeadlessChrome, which is the kind of detail that tells you the maintainers are dealing with sites that block or degrade headless traffic. A sidecar that identifies as a normal browser gets served different content than one that does not.

The v3.7 coverage guard is the other mechanism worth understanding. The README says it recovers pages where extraction kept only a sliver of the body. That is a real failure mode in Readability-style extraction: the algorithm picks a container that turns out to be a caption or a sidebar, and you get a plausible-looking but useless result. A guard that detects thin output and retries is a sensible answer, and the fact that it exists suggests the maintainers hit the problem in practice.

Every conversion gets an 8-hex share id. GET /s/:id returns the cached Markdown and re-fetches from source if the cached copy is older than one hour. The README positions this for subreddit feeds and similar recurring pulls, where a fixed URL should always return current content.

Getting it running and the configuration keys that matter

The README's quick start is four lines. Create a directory, download the compose file from the repository, run docker compose up -d, and the service listens on port 3000. No .env is required because every variable has a default. Pre-built multi-arch images for linux/amd64 and linux/arm64 are on Docker Hub.

The compose file maps ${PORT:-3000} to the container's port 3000 and takes PUBLIC_URL from the environment with a localhost fallback. Beyond that, the README names the keys that change behaviour rather than just listing them.

PULLMD_SOURCE_HEADER is the one to set first. In v3 the body became just a title plus content, with the source URL, fetch date and metadata moved into YAML frontmatter. Setting PULLMD_SOURCE_HEADER=true restores the old inline header. PULLMD_FRONTMATTER_FIELDS trims which frontmatter fields are emitted, which is how you control token spend per conversion.

PULLMD_COVERAGE_GUARD controls the thin-extraction recovery described above. PULLMD_ALLOW_SIGNUP closes self-registration, and scripts/admin.js create-user creates accounts from the shell when it is closed. PULLMD_CACHE_RETENTION_DAYS sets how long cache rows and share links live, defaulting to 90 days, with 0 meaning keep forever.

Optional tiers are opt-in: ?pdf=ocr for the OCR path on table-grade PDFs, and a media tier for image captioning and audio transcription that runs inside the pullmd container against any OpenAI-compatible or local model. Left unconfigured, the README states v3 handles web pages exactly like v2 with a cleaner body.

The v3 clean-body change is the migration cost

The README is direct about this: the clean-body change is the only breaking one in the v2 to v3 upgrade, and MIGRATION.md has a one-line opt-out. Everything else in the 3.x line is described as additive.

Breaking is the right word. If you have a downstream parser that reads the source URL from the top of the Markdown body, it will not find it after upgrading. The data is still there, in frontmatter, but frontmatter is a different parse target. The fix is one environment variable, but you have to know to set it, and you have to know before your pipeline starts producing malformed records.

The upside is real. The README frames the change as token efficiency: nothing is duplicated, and the body is title plus content. For Reddit posts, subreddit, author, upvotes and published date move into frontmatter. If your consumer reads frontmatter, you spend fewer tokens per page. If it does not, you spend the same tokens and lose fields.

PULLMD_FRONTMATTER_FIELDS is the companion control. It is an allowlist, so you can emit only the fields your downstream system actually uses. That is a smaller surface to maintain than parsing a header block, and it is the more durable design. The migration pain is a one-time cost; the frontmatter model is the better long-term shape.

SSRF protection, the sidecar health endpoint and operational limits

Version 3.3 added SSRF protection. The README states that private, loopback, link-local, CGNAT and cloud-metadata targets are rejected by default, on every fetch path and every redirect hop. The redirect-hop detail is the part that matters. A naive blocklist that only checks the initial URL is trivially bypassed by a redirect to 169.254.169.254, and the README explicitly claims coverage at each hop.

If PullMD runs inside a network with internal services reachable from the container, this default is doing load-bearing work. It is also a constraint: any legitimate internal URL you want to convert will be refused, and the README does not describe an allowlist mechanism in the material provided. That is a gap worth confirming before you plan to point it at an internal wiki.

The v3.10 sidecar health endpoint addresses a different operational problem. GET /api/status answers 503 when a configured sidecar stops responding. Without it, a dead Playwright renderer degrades extraction quietly: static pages keep working, JavaScript-heavy pages silently fall back to whatever the earlier stages produce, and you notice weeks later when someone complains about missing content. Turning a silent degradation into an alert is the correct trade.

The PULLMD_CACHE_RETENTION_DAYS default of 90 days is a policy decision you inherit. Cache rows and share links expire together. If you have shared a link externally and expect it to keep resolving, the 90-day default will eventually break it. Setting it to 0 keeps them forever, which is the opposite problem if the fetched content is sensitive.

Where PullMD is the wrong choice, and what to use instead

The licence is the first filter. PullMD is AGPL-3.0. If you plan to offer a modified version as a network service, the AGPL's source-disclosure obligation applies to your modifications. For internal use behind a firewall this is usually a non-issue; for a commercial product built on a fork, it is a decision for your legal team, not for this article.

The second filter is whether you need a renderer at all. If your sources are static HTML, a library like Mozilla Readability called directly from your own code does the extraction without a Chromium sidecar, without a container, and without an HTTP hop. PullMD's value is the cascade plus the format coverage plus the interfaces. If you only need the first stage of the cascade, you are paying operational cost for capability you will not use.

Compare against a hosted conversion API, which is the other obvious alternative. The difference is where the fetch happens and who holds the content. A hosted API means no Docker, no sidecar, no cache retention policy to set, and no SSRF surface to reason about. It also means every URL you convert leaves your network, the vendor sets the rate limits, and the extraction behaviour changes when they change it. PullMD inverts every one of those: you own the pipeline, you own the data, and you own the maintenance. The README's zero-config compose file is genuinely four lines, but a Playwright sidecar and an optional OCR tier are not zero-maintenance over a year.

A purpose-built scraper for one site is the third alternative. If you need Reddit comment trees and nothing else, a small script against the Reddit API is less machinery than a general conversion service. PullMD's Reddit and Hacker News converters are a convenience, not a reason on their own to run the whole stack.

Upgrade cadence and what the release history implies

The release notes show a steady cadence through the 3.x line: 3.1 Hacker News pipeline, 3.2 X-Transcript-Status header, 3.3 SSRF protection, 3.4 query-scoped extraction, 3.5 and 3.6 site recipes including select.content for naming the article body outright, 3.7 coverage guard, 3.8 account controls, 3.9 download button with X-Suggested-Filename, 3.10 sidecar health endpoint, 3.11 configurable cache retention. That is roughly one behavioural addition per minor release.

For an operator, that cadence has a cost. Each minor version can add a configuration key or shift a default, and PULLMD_FRONTMATTER_FIELDS means the set of emitted fields is something you can pin but also something that can grow. Pinning to a specific image tag rather than latest is the straightforward mitigation, and the compose file's image reference is the place to do it.

The query-scoped extraction feature at 3.4 deserves a note because it changes what the service is for. ?query= returns only the sections relevant to a question, with a max_tokens budget. That is a retrieval-shaped feature bolted onto a conversion service, and it implies the maintainers see PullMD sitting in front of an LLM rather than just producing files. Whether that belongs in the same process as the renderer is a design question the README does not address.

The AGPL-3.0 licence is unchanged across the versions listed, and the repository is not archived. The last push and the v3.11.0 release are the same day, which is consistent with active maintenance but tells you nothing about how long that continues.

Editorial conclusion

PullMD fits teams that already run Docker and want one endpoint for web pages, PDFs, images, audio and YouTube transcripts feeding an agent pipeline, especially where Reddit and Hacker News comment trees matter. It is the wrong tool if you cannot accept AGPL-3.0 obligations, if you need a hosted service with no operational surface, or if you only ever convert static HTML and would rather not run a Chromium sidecar. Before adopting, verify three things: that PULLMD_SOURCE_HEADER matches what your downstream parser expects, that your deployment's outbound fetch paths are covered by the SSRF protection defaults, and that PULLMD_CACHE_RETENTION_DAYS suits your retention policy rather than the 90-day default.

Official sources

1. AeternaLabsHQ/pullmd on GitHub
2. Issues
3. License: AGPL-3.0
4. README
5. Releases