AutoCrawler: A Selenium Image Scraper for Google and Naver, and What Breaks When Their DOM Changes

The problem AutoCrawler solves, and for whom

Assembling an image dataset by hand is slow, and assembling one from two search engines with different page structures is slower still. AutoCrawler takes a plain text file of search keywords and produces a directory tree of downloaded images, one folder per keyword, from google.com and naver.com. The README frames the target as deep-learning and bigdata work: the repository topics include deep-learning and bigdata, and the tool exists to turn a keyword list into class-labelled image folders. That framing matters, because the output layout (one directory per keyword) is exactly what an ImageFolder-style loader expects. Naver support is the less common half. Most English-language image scrapers target Google only; AutoCrawler ships a separate collector for Naver, which is useful if your dataset needs Korean-language query coverage. The tool is a command-line program, not a library with a stable API, and it drives a real Chrome browser rather than calling an image search endpoint.

How the crawl actually runs: Selenium, worker processes, and per-site collectors

The architecture visible in the repository is a thin CLI over a set of collectors. The README points to `src/autocrawler/collectors/` containing `google.py`, `naver.py`, and shared helpers in `base.py`, and names four collector functions: `collect_google`, `collect_naver`, `collect_google_full`, and `collect_naver_full`. The split between the plain and `_full` variants mirrors the `--full` flag: thumbnails come from the search results page, full resolution requires visiting each image, which the README describes as slow. Downloading is parallelised with worker processes, controlled by `--threads`, and the README notes that `--proxy-list` assigns each task a randomly chosen proxy from a comma-separated list, which is the only mechanism described for spreading requests across source addresses. Before any worker process starts, the main process opens a single visible Chrome window against Google using a persistent profile at `./chrome-profile`. That warm-up is deliberate: CAPTCHA detection and solving happen in the main process, so the README states you do not need to drop to `--threads 1` to get past a CAPTCHA. Naver is not part of that warm-up, because the README says Naver has not been observed to CAPTCHA-block this crawler. After crawling, the tool compares file counts across directories and reports any directory holding under 50 percent of the average, which is a crude but useful signal that one keyword returned far fewer usable images than the rest.

Getting it running: install, the CLI surface, and the v2.0 flag change

The README's sequence is short. Install Chrome, run `pip install -e .` to install the `autocrawler` package and its dependencies, write your search terms into `keywords.txt`, then run `autocrawler`. The README also accepts `python -m autocrawler` and `python main.py` for backwards compatibility. Output lands in the `download` directory by default. The argument list is where the operational detail lives. `--threads 4` sets the number of worker processes. `--google` and `--naver` are both on by default, so a bare invocation hits both sites unless you pass `--no-google` or `--no-naver`. `--skip` is on by default and skips a keyword whose download directory already exists, which the README describes as necessary when re-downloading; the point being that you must pass `--no-skip` to force a second pass over an existing folder. `--full` is off by default and switches to full-resolution downloads. `--limit 0` means unlimited images per site, so a positive value is your only per-keyword ceiling. `--download-path` and `--keywords-file` override the defaults for the output folder and the keyword list. `--no_gui` accepts `auto`, `true`, or `false`; the default `auto` resolves to false when `--full` is not set and true when it is, and the README warns headless mode is an acceleration for full-resolution mode but unstable in thumbnail mode. The README flags one breaking change: as of v2.0 the boolean flags use `--flag` and `--no-flag` pairs instead of `--flag true` and `--flag false`, while `--no_gui` keeps its old form. Any script written against the pre-2.0 syntax needs editing. For headless servers the README gives a virtual-display recipe: install `xvfb` and `screen`, start a screen session, then `Xvfb :99 -ac & DISPLAY=:99 autocrawler`.

XPath selectors are the maintenance surface, and they will break

This is the central design trade-off. AutoCrawler drives the live search pages through Selenium and locates elements with XPath, so any redesign of Google's or Naver's image results can silently break collection. The README says this plainly: the sites change consistently, and you may need to fix the XPath selectors in `src/autocrawler/collectors/google.py` or `naver.py`. It then walks through the debugging loop: open the image search URL, open Chrome developer tools, inspect the target image element, open the collector file, and test candidate XPath expressions with Ctrl+F in the developer tools. There is no selector configuration file and no plug-in registry; fixing a break means editing Python source. The README also records that as of 2026-08 all four collectors were re-verified end-to-end against the live DOM and that several selector and timing bugs were fixed, with a `NOTE` comment at the top of each function describing what changed and why. Treat that as a snapshot, not a guarantee. A crawler verified against a live DOM in August 2026 is one layout change away from returning zero images, and the failure mode is quiet: you get an empty or short directory rather than an exception. The 50 percent imbalance report is the closest thing to a canary, and it only fires after a run completes.

CAPTCHA handling puts a human in the loop

Google's bot detection is described in the README as IP-reputation based rather than purely per-browser, which has a practical consequence: a burst of automated requests, including ordinary testing and debugging, can flag your address for a while, and a fresh browser process may be challenged again even with valid cookies. AutoCrawler's answer is a pre-flight step. Before crawling, it opens one visible Chrome window against Google using the persistent profile at `./chrome-profile`. If a CAPTCHA appears, the terminal prints a message naming the `https://www.google.com/sorry/index?...` URL and instructs you to solve the challenge in the Chrome window and press Enter. The detector polls for a few seconds before giving up, because the CAPTCHA widget can take a moment to render. Two things follow from this. First, unattended runs are not reliable against Google: someone has to be present to solve the challenge, and a scheduled job that hits one will stall or fail. Second, the persistent profile is part of the mechanism, so deleting `./chrome-profile` between runs discards whatever trust state it accumulated. Naver is explicitly excluded from the warm-up, which means a Naver-side block, if it ever appears, has no handling path in this codebase.

The macOS chromedriver problem is a separate failure mode

The README documents a hang that has nothing to do with the crawler logic. On macOS, `chromedriver` may hang indefinitely, including when run standalone outside this project, or end up stuck and unkillable. The stated cause is Gatekeeper rejecting unnotarized, ad-hoc-signed binaries. The diagnostic given is `spctl -a -vv "$(which chromedriver)"`; if it prints `rejected`, the binary is not trusted. The README notes that on macOS 15 and later, `spctl --add` no longer works and reports `This operation is no longer supported`, leaving `sudo spctl --global-disable` as the remaining override, which surfaces an allow-anywhere toggle plus an Open Anyway confirmation. It also reports that in practice simply running `chromedriver` once directly in an interactive Terminal, not through a script, got it trusted without touching that toggle. There is a second wrinkle worth reading twice: `build_driver()` prefers a `chromedriver` already on PATH over downloading a separate copy via `webdriver-manager`, but each distinct binary path needs its own Gatekeeper approval, so a copy under `~/.wdm/` is a different file from one on PATH and must be approved separately. If you install via Homebrew and still see rejections inside the project, that PATH-versus-`~/.wdm/` distinction is the first thing to check. Note that `sudo spctl --global-disable` is a system-wide security setting change, not a project-level fix.

Where AutoCrawler is the wrong tool, and what to use instead

AutoCrawler is the wrong choice when you need a reproducible dataset build. It has no manifest of downloaded URLs, no deduplication step described in the README, and no pinned record of which page layout produced a given file. Two runs a month apart can return different sets, and the only integrity signal is the post-run imbalance report. If your pipeline needs to rebuild a dataset from a pinned snapshot, a crawler that parses live DOM is the wrong layer. The closest alternative in approach is a search-API client, for example a library that queries the Google Custom Search JSON API or the Naver Search API and downloads image URLs from structured responses. The difference is architectural, not cosmetic: an API client parses JSON with a documented schema, so a field rename is a versioned change you can read about, while AutoCrawler parses rendered HTML with XPath you must re-derive by hand in developer tools. The trade-offs run the other way too. API clients are subject to quota limits and result caps that the API imposes, and Naver's image coverage through its API may not match what the consumer search page returns. AutoCrawler has no such documented quota; its limit is Google's CAPTCHA and your patience. If you need breadth from the consumer-facing result pages and can absorb selector maintenance, AutoCrawler is the more direct route. If you need something you can schedule without a human watching for a challenge, the API route is the one to evaluate.

Licence, maintenance cost, and what to check before you commit

AutoCrawler is Apache-2.0, which permits commercial and private use and includes an explicit patent grant, with the usual requirements around preserving notices and stating changes. That covers the code. It does not cover the images: the README says nothing about the terms of service of Google or Naver, about robots.txt, or about the copyright status of what you download. Downloading images from a search results page is a separate question from the licence on the crawler, and the repository gives you no guidance on it. Treat the Apache-2.0 grant as applying to `src/autocrawler/` only. On maintenance cost, the honest estimate is that selector drift is ongoing work, not a one-time setup. The README's own instructions assume you will edit collector source, and the 2026-08 re-verification note shows the maintainers doing exactly that. The codebase is small enough that this is tractable for one engineer: two site collectors, a shared base module, and a CLI. It is not tractable if nobody on your team can read XPath and Chrome developer tools. There are no releases retrieved for this repository, so version pinning is by commit rather than by tagged version, which complicates the v2.0 flag migration if you are copying configuration from older examples.

Editorial conclusion

Adopt AutoCrawler if you need labelled image folders per keyword from Google or Naver and you are willing to maintain XPath selectors yourself. Do not adopt it if you need reproducible dataset builds, a stable API, or unattended long-running collection: Google's IP-reputation CAPTCHA handling and the macOS chromedriver trust problem both require a human in the loop. Before committing, verify three things on your own machine: that `spctl -a -vv "$(which chromedriver)"` does not print `rejected`, that the four collectors in `src/autocrawler/collectors/google.py` and `naver.py` still match the live DOM after the 2026-08 re-verification, and that your intended use of the downloaded images is permitted by the source sites' terms, which the README does not address.

Official sources

1. Issues
2. License: Apache-2.0
3. README
4. YoongiKim/AutoCrawler on GitHub