# website-scraper: pulling a whole site to a local directory, assets included

> website-scraper downloads a site, with its stylesheets, images and scripts, into a directory on your machine, following hyperlinks and rewriting URLs as it goes. It parses HTTP responses rather than running JavaScript, so a page that renders client side comes out empty, and a separate puppeteer-based package exists for that case. It is MIT licensed and pure ESM.

**website-scraper/node-website-scraper** — Download website to local directory (including all css, images, js, etc.)

- Repository: https://github.com/website-scraper/node-website-scraper
- Website: https://www.npmjs.com/package/website-scraper
- Stars: 1,762 · Forks: 305
- Language: JavaScript
- License: MIT
- Published: 2026-10-19 · Updated: 2026-10-19 · Language: en
- Canonical page: https://hysenlabs.com/projects/website-scraper-node-website-scraper

## What it does, and the one limitation that shapes everything

The package is described as downloading the website to the local directory, including all css, images, js and so on, and the README states the limitation in a note directly under that line. By default, dynamic websites, where content is loaded by JavaScript, may not be saved correctly, because website-scraper does not execute JavaScript, it only parses HTTP responses for HTML and CSS files. If you need to download a dynamic website, the README points to a separate project, website-scraper-puppeteer. That single sentence tells you what the library is and is not. It is an HTTP client with an HTML parser, not a browser. Everything it produces comes from bytes the server sent, which is why it is fast and why it misses anything assembled on the client.

## The API is one function with an options object, and it is promise based

The usage section is short. You import scrape as an ES module, pass an options object with the urls to download and the directory to save them into, and await the result, or use it as a promise. The two required options are urls and directory, and everything else has a default. The defaults are published as a separate export, so you can read them at runtime with an import of website-scraper/defaultOptions, and they also live in lib/config/defaults.js in the repository. The plugin system is the extension point, and the options list is long enough to be worth reading as a whole: sources to select which resources to download, recursive to follow hyperlinks in HTML files, maxRecursiveDepth and maxDepth, request options passed through to got, subdirectories for file extensions, defaultFilename, prettifyUrls, ignoreErrors, urlFilter, filenameGenerator, requestConcurrency and plugins. The existence of a plugins option is what keeps this from being a single-purpose script.

```javascript
import scrape from 'website-scraper'; // only as ESM, no CommonJS
const options = {
  urls: ['http://nodejs.org/'],
  directory: '/path/to/save/'
};

// with async/await
const result = await scrape(options);
```

## urls and directory, and the rule about existing directories

The urls option is an array of objects containing the URLs to download and the filenames for them, and a plain string is shorthand for the default filename of index.html, so you can mix a bare URL with an object that names the file. The directory option is an absolute path to where the files are saved, and the requirement that catches people is that the directory should not exist, because it will be created by the scraper. The README links to a FAQ entry explaining how to download into an existing directory and why it is not supported by default, so the behaviour is deliberate rather than an oversight. The reasoning behind it is the sensible one, since a scraper that merges into a directory it did not create is a scraper that can overwrite files it did not write, and the same question of where the output goes comes up for every option that produces a file.

## sources, selectors and the cheerio dependency

By default the scraper tries to download all possible resources, and the sources option lets you narrow that with selectors and attribute values. Each source is an object with a selector and an attribute, and the example picks images with an img selector and the src attribute, stylesheets with a link selector for rel equals stylesheet and the href attribute, and scripts with a script selector and the src attribute. The selector is parsed with cheerio, so it can be any selector cheerio supports, which is what makes the option general rather than a fixed list of resource types. The cheerio dependency also explains the general design, since cheerio is an HTML parser and a jQuery-like selector engine over it, so a resource that is not referenced by a standard tag or attribute is invisible to the scraper unless you write a selector for it.

```javascript
// Downloading images, css files and scripts
scrape({
  urls: ['http://nodejs.org/'],
  directory: '/path/to/save',
  sources: [
    {selector: 'img', attr: 'src'},
    {selector: 'link[rel="stylesheet"]', attr: 'href'},
    {selector: 'script', attr: 'src'}
  ]
});
```

## Recursion and depth, the two limits that stop a scrape running away

The recursive option makes the scraper follow hyperlinks found in HTML files, and the README's warning is the practical one, to remember to set maxRecursiveDepth to avoid infinite downloading. It defaults to false. maxRecursiveDepth is a positive number giving the maximum depth for hyperlinks, and the detail is that other dependencies are saved regardless of their depth, so it bounds the link following and not the asset downloading. maxDepth is the broader limit, a positive number for the maximum depth of all dependencies, and both default to null, meaning no maximum is set. requestConcurrency sets the maximum concurrent requests, which is the other lever that stops a crawl from becoming a flood. These options together are what a careful user of the library tunes first, because without them the default behaviour on a site with a nav bar is to walk a very large graph.

## The package internals, and what the dependency list tells you

The manifest is small and each dependency answers a question. cheerio is the HTML parser and selector engine, got is the HTTP client, and the request option is described as custom options for the got module, so the HTTP layer is replaceable. normalize-url handles the prettifyUrls option, css-url-parser and srcset deal with CSS urls and the srcset attribute, filenamify is what turns a URL into a safe filename, and p-queue is the concurrency limiter behind requestConcurrency. debug is the log and debug mechanism. The package is pure ESM, with a type of module and an exports map pointing at index.mjs and the defaultOptions subpath, and the engines field requires Node 20.18.1 or newer. The README is explicit that website-scraper since v5 is pure ESM and does not work with CommonJS, with a link to the v5.0.0 release notes, which is the single most likely thing to trip up an adopter.

## Testing, the migration note and the release history

The test setup has three layers. The unit and functional tests run under Mocha with c8 for coverage, the end to end tests have their own script with a five minute timeout because they hit the network, and the eslint script covers lib and the entry point. The dev dependencies include nock, which is how the HTTP layer is faked in the tests, and sinon. The repository has a MIGRATION.md, which is what you would expect given the v5 module format change and the v6 release that followed. The release history is v5.3.0 and v5.3.1 in 2022, then a long gap and v6.0.0 on 2025-10-30, so the sixth major version is recent and the three years before it were quiet. The last push to the repository was on 2026-10-13, the repository is not archived, and the licence is MIT with the author recorded as Sofiia Antypenko.

## Where this fits, and the honest use case

The sensible reading is that this is a library for archiving and offline reading of sites you control or have permission to copy, and for generating a static copy of a site that renders server side. It is not a crawling framework, and it is not a browser automation tool, and the two facts are the same fact seen from two sides. The FAQ document, the contributing guide and the code of conduct are all in the repository, and the project is sponsored through GitHub Sponsors with a list of backers in BACKERS.md. If you need a dynamic page, the answer is the puppeteer variant, and if you need a distributed crawler with politeness, scheduling and a queue, this is the wrong size of tool.

## Conclusion

website-scraper suits someone who needs a copy of a site on disk, for archival, for offline reading or for feeding a static site, and who controls the sites being downloaded. It is the wrong choice for a site whose content is produced by JavaScript, because the library parses responses rather than executing a browser, and a poor fit if you need politeness controls or a distributed crawler, since it is a library for one site at a time rather than a crawler with scheduling and rate limiting. Before adopting it, check the ESM constraint, since version 5 and later are pure ESM and do not work with CommonJS, and read the FAQ entry about the target directory, which must not already exist and is created by the scraper.

## FAQ

### What does website-scraper download?

It downloads a website to a local directory, including the CSS, images and JavaScript, following hyperlinks in the HTML files when recursive is enabled and rewriting URLs as it goes. The two required options are urls, the pages to download, and directory, the absolute path to save them into.

### Why does website-scraper fail on JavaScript-heavy sites?

Because it does not execute JavaScript. It only parses HTTP responses for HTML and CSS files, so content assembled on the client is never present in the responses it reads. The README points to a separate project, website-scraper-puppeteer, for downloading dynamic websites.

### Can I use website-scraper from CommonJS?

No. The README states that website-scraper since v5 is pure ESM and does not work with CommonJS, and the import is written as ESM only. The package requires Node 20.18.1 or newer, and the v6.0.0 release followed v5.3.1 from 2022 after a long gap.

### How do I limit how much website-scraper downloads?

With maxRecursiveDepth, a positive number for the maximum depth of hyperlinks, with other dependencies saved regardless of depth, and maxDepth, a positive number for the maximum depth of all dependencies. Both default to null, meaning no maximum is set, and requestConcurrency limits how many requests are in flight at once. The README warns to set maxRecursiveDepth to avoid infinite downloading.

### Can website-scraper save into a directory that already exists?

Not by default. The directory option must not exist, and the scraper creates it. The README links to a FAQ entry that explains how to download into an existing directory and why that is not supported by default.

## Sources

- [License: MIT](https://github.com/website-scraper/node-website-scraper/blob/master/LICENSE)
- [Project website](https://www.npmjs.com/package/website-scraper)
- [README](https://github.com/website-scraper/node-website-scraper/blob/master/README.md)
- [Releases](https://github.com/website-scraper/node-website-scraper/releases)
- [website-scraper/node-website-scraper on GitHub](https://github.com/website-scraper/node-website-scraper)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/website-scraper-node-website-scraper