page-fetch: archiving a page and everything it loads, with a headless browser
Fetch web pages using headless Chrome, storing all fetched resources including JavaScript files. Run arbitrary JavaScript on many web pages and see the returned values
At a glance
- What is it?
- page-fetch is a small MIT-licensed Go tool that drives headless Chrome through chromedp, saves every resource a page requests into a directory that mirrors the site's own structure, and can run arbitrary JavaScript on each page and print the results. It suits a researcher archiving a site or inspecting what a page computes, and it needs a local Chrome or Chromium install.
- Who is it for?
- page-fetch is worth using if you need to keep a copy of what a page actually loaded rather than what the server returned, because a browser-driven fetch captures the JavaScript, the stylesheets and the fonts that a plain HTTP client never sees, and because the output directory mirrors the site layout, so the archive is browsable. It is the wrong tool if you only need the HTML, and it is the wrong tool if you need to fetch thousands of pages quickly.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 112 days ago.
- What is it written in?
- Mainly Go, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on October 20, 2026, and from our analysis. They are not legal advice.
Editorial analysis
What it does, and the two jobs it does well
page-fetch is a tool for researchers, and the readme gives it two jobs.
The first is fetching web pages using headless Chrome, storing all fetched resources including JavaScript files. The second is running arbitrary JavaScript on many web pages and seeing the returned values.
The first of those is the more interesting one, and the reason is what a headless browser gives you that a HTTP client does not. A page that renders its content with JavaScript returns an empty shell to a plain client, and the resources that make it work are fetched by the browser as a side effect of running the page. A tool that saves every one of those requests captures the page as a browser saw it, and the output directory then contains the HTML, the scripts, the stylesheets and the images, in a layout that mirrors the original site.
The second job is what turns an archiver into an inspection tool. Since the browser has already executed the page, the JavaScript environment is live, and a script can be evaluated in it and its result printed. That means you can ask a page a question, for example what its document domain is, or what the href of every link is, and get an answer per page, without writing a parser for the page's own data structures.
The tool is written in Go, the licence is MIT, the default branch is master, and the last push to the repository was on 2026-07-03. There are no GitHub releases, and the whole program is a single main.go with a go.mod and a go.sum, so the codebase is small enough to read in one sitting.
Installing it, and the browser it looks for
The install is the standard Go install:
▶ go install github.com/detectify/page-fetch@latestor you clone the repository and build it manually:
▶ git clone https://github.com/detectify/page-fetch.git
▶ cd page-fetch
▶ go installThe dependency is chromedp, the Go binding for the Chrome DevTools Protocol, and that brings a requirement the readme states clearly: a Chrome or Chromium browser has to be installed. The tool does not bundle a browser.
What it does do is search for one, and the list of executable names it tries in order is worth knowing because it is the difference between working and a confusing failure:
headless_shell
headless-shell
chromium
chromium-browser
google-chrome
google-chrome-stable
google-chrome-beta
google-chrome-unstable
/usr/bin/google-chromeThe headless shell entries come first, which is the right order, because the headless shell is a much smaller binary than a full Chrome and the tool is only ever going to drive it headlessly. The two headless spellings are there because the binary was renamed between Chrome versions, and the beta and unstable entries are there so that a developer testing a new Chrome does not have to reinstall their stable browser.
The dependency set in the go.mod file is correspondingly small. The direct requirements are the chromedp client, the cdproto package with the protocol definitions, and golang.org/x/net, with the indirect list being chromedp's own utilities and the gobwas libraries for header handling, a connection pool and WebSockets, plus the JSON and interning helpers the protocol needs.
That dependency list is also the clearest description of how the tool works. It speaks the DevTools Protocol to a browser over a WebSocket, it subscribes to the network events the browser reports, and it writes the responses to disk.
Feeding it URLs, and what comes out
The input is a list of URLs on standard input, so the tool composes with everything else in a shell.
▶ page-fetch < urls.txtOr using the output of another command:
▶ grep admin urls.txt | page-fetchBy default, responses are stored in a directory called out, which is created if it does not exist, and each request is echoed with its status and content type as it happens:
▶ echo https://detectify.com | page-fetch
GET https://detectify.com/ 200 text/html; charset=utf-8
GET https://detectify.com/site/themes/detectify/css/detectify.css?v=1621498751 200 text/css
GET https://detectify.com/site/themes/detectify/img/detectify_logo_black.svg 200 image/svg+xml
GET https://fonts.googleapis.com/css?family=Merriweather:300i 200 text/css; charset=utf-8The directory structure used in the output directory mirrors the directory structure used on the target websites, and the tree in the readme shows it, with the site name as the top directory, then the path within the site, and the file names without an extension:
out
├── detectify.com
│ ├── index
│ ├── index.meta
│ └── site
│ └── themes
│ └── detectify
│ ├── css
│ │ ├── detectify.css
│ │ └── detectify.css.metaAnd a .meta file is stored for each request, containing the originally requested URL including the query string, the request and response headers, and so on.
The .meta sidecar is a good design. It means the archive is a set of plain files you can open, grep and diff, and the metadata that would otherwise be lost, the query string that produced a variant of a page, the response headers, the status code, is kept next to the content rather than in a database somewhere.
The output directory can be renamed with the -o and --output option, and the directory is created if it does not already exist.
Running JavaScript on each page
The JavaScript option is the one that makes this more than a scraper.
You can run arbitrary JavaScript on each page with the -j and --javascript option. The return value of the JavaScript is converted to a string and printed on a line prefixed with JS:
▶ echo https://example.com | page-fetch --javascript document.domain
GET https://example.com/ 200 text/html; charset=utf-8
JS (https://example.com): example.comThe readme notes that this option can be used for a very wide variety of purposes, and gives the example of extracting the href attribute from all links on a webpage:
▶ echo https://example.com | page-fetch --javascript '[...document.querySelectorAll("a")].map(n => n.href)' | grep ^JS
JS (https://example.com): [https://www.iana.org/domains/example]The output format is worth noticing, because it is designed to be machine-read. Each result is a single line prefixed with JS and followed by the URL in parentheses, so the JS output is interleaved with the GET lines but can be separated with a grep, as the example does. That matters when you are running the tool over a list of hundreds of pages and want one answer per page rather than a screen of scrolling output.
The script runs in the page's own JavaScript context, so it has the DOM of the rendered page rather than the source HTML, and the result is stringified, which means an object comes back as its JavaScript representation rather than as JSON, and a promise is not awaited unless the script awaits it itself.
That last point is a real limitation for anyone who wants to scrape data from a page that fetches it asynchronously. The script is evaluated once the page has loaded, so anything the page has already computed is available, and anything it would compute later is not unless the script itself waits, and the printed result is a string either way.
The options, and the two that decide what an archive contains
The help output lists the whole option set, and two of them deserve a paragraph of their own.
Options:
-c, --concurrency <int> Concurrency Level (default 2)
-d, --delay <int> Milliseconds to wait between requests (default 0)
-e, --exclude <string> Do not save responses matching the provided string (can be specified multiple times)
-i, --include <string> Only save requests matching the provided string (can be specified multiple times)
-j, --javascript <string> JavaScript to run on each page
-o, --output <string> Output directory name (default 'out')
-p, --proxy <string> Use proxy on given URL
-w, --overwrite Overwrite output files when they already exist
--no-third-party Do not save responses to requests on third-party domains
--third-party Only save responses to requests on third-party domainsThe content type filters are the pair that controls the size of the archive. The -e and --exclude option drops any response whose content type partially matches the given value, and the documented example is to avoid storing image files by specifying image/. The -i and --include option does the opposite, keeping only the content types that match, and the example keeps text/html. Both can be given more than once, so a run can keep HTML and CSS and nothing else.
The third-party pair is the one with the most effect on what you end up with. The --no-third-party option drops responses to requests for domains that do not match the input URL or one of its subdomains, and the --third-party option does the opposite and keeps only those. The readme describes both as things you might occasionally want, and for a researcher they are more than occasional, because by default the tool keeps the analytics, the tag manager and the advertising scripts that every site loads, and those are the files nobody wants in an archive.
The remaining options are straightforward. Concurrency sets how many headless Chrome processes are used, with a default of 2, which is a conservative default and the right one for a tool that will happily launch a browser per request. The delay option waits between requests in milliseconds, which is what you need to be polite to a server. The proxy option points all requests at a proxy, and the documented example is the Burp Suite proxy on localhost. The overwrite option changes the file naming, because by default when a file already exists a new file is created with a numeric suffix, so index becomes index.1.
Editorial conclusion
page-fetch is worth using if you need to keep a copy of what a page actually loaded rather than what the server returned, because a browser-driven fetch captures the JavaScript, the stylesheets and the fonts that a plain HTTP client never sees, and because the output directory mirrors the site layout, so the archive is browsable. It is the wrong tool if you only need the HTML, and it is the wrong tool if you need to fetch thousands of pages quickly. Before you rely on it, check three things: that a browser is available, since the tool needs a Chrome or Chromium install and it searches a fixed list of executable names before giving up; that you understand what the arbitrary JavaScript option does, because the script runs on every page you feed it and its return value is printed, which is exactly the power you want and exactly the power you should only point at pages you intend to execute code on; and that you set the third-party flags for your use, because by default the tool saves everything a page loads, including analytics and advertising scripts from other domains, which is rarely what a researcher wants to keep.
Frequently asked questions
What does page-fetch store, and where?
Responses are stored in a directory called out by default, and the directory structure mirrors the structure of the target website. A .meta file is stored for each request, containing the originally requested URL including the query string along with the request and response headers. The output directory name can be changed with the -o and --output option, and it is created if it does not exist.
How do I run JavaScript on the pages page-fetch visits?
Use the -j and --javascript option. The return value of the script is converted to a string and printed on a line prefixed with JS and the page URL, so the output can be separated from the request log with a grep. The script runs in the rendered page, so it sees the DOM rather than the source HTML, and it can be used to extract values such as document.domain or every link href.
What does page-fetch need installed besides itself?
A Chrome or Chromium browser, because the tool drives it through chromedp and does not bundle one. It searches for the executable in order, starting with headless_shell and headless-shell, then chromium and chromium-browser, then the google-chrome variants including the beta and unstable builds, and finally the absolute path to the Chrome binary. The tool is installed with go install and requires Go 1.20 or newer.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/detectify-page-fetch)Was this useful?