# DIYSearchEngine: a Go search engine with a tutorial attached

> DIYSearchEngine is an open source web search engine written in Go, and the unusual part is that the tutorial is the primary artefact: a long walk through building the crawler, the inverted index and the BM25 ranking, with the numbers from a real crawl of a hundred million URLs. The engine is split into four processes over MySQL and Redis, which is where the scaling story lives.

**johnlui/DIYSearchEngine** — 🔍 Go 开发的开源互联网搜索引擎，附教程《自己动手开发互联网搜索引擎》

- Repository: https://github.com/johnlui/DIYSearchEngine
- Website: https://pphc.lvwenhan.com/tech-epic/2023/diy-search-engine
- Stars: 704 · Forks: 94
- Language: Go
- License: not declared
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/johnlui-diysearchengine

## The tutorial is the product, and it comes with a legal warning before the first line of code

DIYSearchEngine is described as an open source internet search engine developed in Go, capable of collecting large amounts of internet data at high speed, and the project is paired with a long tutorial titled roughly a twenty thousand word guide to building your own internet search engine.

The tutorial opens with a warning that is worth taking seriously, because it is unusual to see it this prominently. It is an article about building a legal search engine, everything complying with the requirements of the Cybersecurity Law and with the robots industry conventions, and the author suggests that if your company asks you to crawl sites that have anti-scraping measures, you should leave immediately, having heard of cases where an entire company of several hundred people was taken down at once.

That framing sets the tone for the whole project. The argument in the tutorial is that search engine articles are everywhere and they all follow the same three-part formula of crawler, index and ranking, but the parts that matter, how to actually write them, are kept quiet. The response is a follow-along article, step by step, ending with a search engine you can run.

The stated goal is that the data volume is not as large as you imagine, the inverted index is not as exotic as it looks, and BM25 is not as dramatic as its formula suggests. For a handful of users there is not much computational pressure at all.

The legal note and the technical note are the same note. A crawler that ignores robots.txt is not a fast crawler, it is a liability, and the project takes the compliance side seriously enough to put it in the first paragraph and then implement it in the code.

## The stack: Go, Redis for coordination, MySQL for the dictionary

The backend language is Go, the in-memory database is Redis, and the dictionary storage is MySQL. The tutorial is explicit about why this combination avoids work: you do not have to spend effort on inter-process communication, you do not have to solve multithreading and thread-safety problems, and you do not have to hand-build a densely packed B+ tree on disk.

The dependency list in go.mod shows how thin the surface is, and the module is named github.com/johnlui/enterprise-search-engine even though the repository is DIYSearchEngine, which tells you the project was renamed at some point.

The direct dependencies divide into four groups. The parsing and storage side is PuerkitoBio/goquery for HTML parsing, gorm.io/gorm with the MySQL and SQLite drivers for the database, and golang.org/x/text for text handling. The coordination side is go-redis/redis/v8 and github.com/yanyiwu/gojieba, which is the Chinese word segmenter, and that last one is the piece that makes the whole thing work for Chinese content, because an inverted index over Chinese text needs a word splitter, and jieba is the standard choice. The plumbing is joho/godotenv for reading the .env file, robfig/cron/v3 for scheduling, and gin-gonic/gin for the HTTP layer. The test side has alicebob/miniredis/v2, an in-process Redis implementation, which is how you test code that talks to Redis without running a Redis server.

The Go version declared in go.mod is 1.19, which is a sensible floor for a project that will be cloned and built by people learning from it, and the Docker build uses a newer toolchain, golang 1.22 on bookworm, so the two are not tied together.

The scale of the whole thing is stated as a data point rather than a goal. The author says the crawler took 2 months to reach about 100 million URLs, of which 36 million had their HTML text stored in the database, consuming more than 600GB of disk.

## The crawler, and the hardware arithmetic that justifies Redis

The crawler is the first of the three steps, and the tutorial's argument for goroutines is blunt: Go's coroutines make it particularly suitable for high performance crawlers, and with Redis handling inter-coroutine communication, however many CPU cores you have can be used up, while the code stays simple and neither processes nor threads need to be managed by hand.

The cost of that convenience is stated as well. Because coroutines are capable and the code is short, the debugging cost is high, and the author describes the feeling of writing coroutine code as alchemy, where changing one character can take the program from glacial to ten thousand times faster.

The database is a single table, and the schema is worth reading because it is a URL decomposed into its parts. The pages table stores the id, the url, the host, two domain suffix columns, path, query, scheme, the title, the page text, a flag for whether the page has been split into the dictionary, a flag for whether it has been crawled, a crawl timestamp, the original title text of the referring link, the referrer id, and creation timestamps. Two columns exist purely to make the crawler resumable, dic_done and craw_done, which is the difference between a crawler you can restart and one you cannot.

The crawl loop is a recursion, and the guide is careful to say that recursion is being used for the first time in the example. Each round takes the next 100 URLs where craw_done is zero, ordered by id, fetches each page with a curl-style helper, parses out the title and the links, fills in the domain, path and timestamp fields, and then inserts the discovered links back into the pages table, which is how the corpus grows.

The hardware arithmetic is the part that makes the Redis dependency concrete. The Chinese web has 4 million websites and 350 billion pages, and even crawling a thousandth of that is 350 million pages. If you assume 20KB per page uncompressed, 100 million pages needs 20 * 100000000 / 1024 / 1024 / 1024 = 1.86TB of disk, and storing the text in MySQL needs more. The author measured 600GB for 36 million stored pages, which is consistent with that estimate.

Single threaded and blocking, that would take years. The stated hardware requirement is one physical machine with as many cores as possible for the ese binary, plus high performance databases, 16 cores and 64GB with NVMe, and as many database machines as you can get, with the claim that even 65536 databases could be filled given that web data is discrete and its relationships only matter at query time.

## The four process split, and the environment variables that drive it

The Docker Compose file is the clearest statement of the architecture, because it runs one binary in four different roles.

The common block sets the environment shared by all of them, and the services differ only in the command they run. The ese-server runs ./ese serve and publishes port 8080, the ese-scheduler runs ./ese scheduler, the ese-crawler runs ./ese crawler, and the ese-indexer runs ./ese indexer. MySQL 8.4 and Redis 7.2 are the two backing services, each with a named volume, and each of the four application services depends on both.

That split is the design. The crawler, the indexer, the scheduler and the web server have wildly different resource profiles, and giving each its own process means you can scale the crawler with more machines without touching the web tier, and restart the indexer without dropping requests.

The compose file spells out the four roles as one command each:

```yaml
  ese-server:
    command: ["./ese", "serve"]
    ports:
      - "8080:8080"

  ese-scheduler:
    command: ["./ese", "scheduler"]

  ese-crawler:
    command: ["./ese", "crawler"]

  ese-indexer:
    command: ["./ese", "indexer"]
```

The tuning knobs are all in the environment, and the .env.example lists them as a flat list:

```bash
CRAWL_BATCH=4
CRAWL_WORKERS=64
PREPARE_BATCH=20
INDEX_PAGES_PER_SHARD=2
INDEX_TRANSFER_BATCH=2000
INDEX_TRANSFER_WORD_DEPTH=10000
```

CRAWL_WORKERS at 64 is the concurrency of the crawl stage, and the three INDEX_ variables control how the index build moves data from the dictionary into the inverted index. The connection pools are configurable too, with DB_MAX_IDLE_CONNS at 1 and DB_MAX_OPEN_CONNS at 20, the same pair again for the dictionary database as DB_DIC_MAX_IDLE_CONNS and DB_DIC_MAX_OPEN_CONNS, and REDIS_DIAL_TIMEOUT_SECONDS and REDIS_READ_TIMEOUT_SECONDS at 1 each.

An idle connection count of 1 is a strange number to write down on purpose, and it is the kind of detail that shows the author tuned these against a real workload rather than accepting a library default.

The two database configurations are separate hosts and separate databases, DB_HOST0 for the crawled pages and DB_HOST_DIC for the dictionary, so the two halves of the system can live on different machines, which is what makes the sharding story possible.

## robots.txt handling, and the user agent question it raises

The crawler checks permissions before it fetches, and the mechanism is specific. The project uses a robotstxt library to find out whether the crawler is allowed to fetch a given URL, stores the rules for each domain in a separate table, and keeps a cache in Redis, so that before every URL is fetched there is a match, and the fetch only happens if the match succeeds.

Caching the rules in Redis and the authoritative copy in MySQL is the right split for a crawler that will ask the same question about the same domain thousands of times in a row, and it is the same reason Redis appears in the index stage as well.

The user agent is a separate decision, and the tutorial walks through it as a design step. The guidance is to base it on a recent desktop browser user agent and modify it to include your own spider name. The example given uses a Chrome user agent with ESESpider/1.0 appended, where ESE is short for Enterprise Search Engine, the name the author gave the project.

And then the caveat, which is the part that is honest about the limits of any crawler. Some sites block crawlers that are not the major search engines, and the advice on that is, in the guide's phrasing, that you will need to solve it yourself. That is the point where the legal framing and the technical one meet: the rules say respect robots.txt, and the practical note says that is not the same as being allowed in, and the difference is a legal question, not a technical one.

The project structure shows where that work lives. There is a crawl_internal.go for the crawler, a cron.go for the scheduler, a dict/ directory for the dictionary stage, a db/ directory, an internal/ directory, models/, controllers/, tools/, views/ and a config/ directory. The presence of a Dockerfile and a docker-compose.yaml means the whole four-process system starts with a single compose file.

## Maintenance, the missing licence, and what to check before you clone it

The maintenance picture is thin in a way that matches the project's nature. The release list has two entries, v1.0.0 on 2023-07-04 and v1.0.1 on 2023-07-06, both from the same week in July 2023, and the last push to the repository was on 2026-08-27 with the repository not archived.

So the source receives occasional work while the published version has not moved since 2023. For a tutorial project that is close to ideal, because the code you read is the code the tutorial describes, and there is little churn to follow.

The licence is the item to check. The platform records the licence as unknown, and there is no LICENSE file at the repository root, which is unusual for a project of this size. That is a real gap rather than a formality, because the code is a complete search engine implementation and a tutorial that is explicitly framed as something you follow step by step to build your own. Whether the terms permit that depends on terms the repository does not state, so before you copy code out of it into a project of your own, you have to ask the author.

The homepage is a blog post rather than a documentation site, at pphc.lvwenhan.com, and the top of the guide points readers to the bottom of the project for a tutorial on how to run it, which suggests the running instructions are in a separate section of the long article rather than in the repository.

There is also a Dockerfile that reveals a build constraint worth knowing. The build stage installs g++ because gojieba needs its CGO part compiled, and the runtime stage is debian:bookworm-slim rather than Alpine, with a comment explaining that the CGO and gojieba binary would fail to run in a musl environment. That is a specific, hard-won detail: a Go project with a CGO dependency is effectively tied to a glibc base image, and the Dockerfile says so rather than leaving you to discover it.

The entrypoint is the multi-role binary again, CMD ["./ese", "all"], which runs every stage in one container, and the compose file overrides it per service to run one stage each.

## Conclusion

Adopt DIYSearchEngine if you want to read how a search engine is actually put together and want runnable code alongside the explanation, because the tutorial and the code are the same project here. Do not adopt it as a search backend you will operate, since it is one person's teaching implementation with an unknown licence, and the crawling half needs your own compliance work. Before you run anything, read the legal framing at the top of the tutorial, and check the licence situation, because the platform records none for the repository.

## FAQ

### What is DIYSearchEngine and what does it cover?

It is an open source internet search engine written in Go, paired with a long tutorial on building one yourself. The three steps it covers are writing a high performance crawler, splitting pages into a dictionary with an inverted index, and returning results with the BM25 algorithm.

### What databases and libraries does DIYSearchEngine use?

Go for the backend, Redis as the in-memory database and MySQL for dictionary storage. The direct dependencies include goquery for HTML parsing, gorm with the MySQL and SQLite drivers, go-redis, gojieba for Chinese word segmentation, godotenv, robfig/cron, gin, and miniredis for testing.

### How does the crawler stay within the rules set by site owners?

It checks robots.txt before every fetch using a robotstxt library, stores the rules for each domain in a separate table, and keeps a cache in Redis so the check is cheap. The tutorial frames the whole project as building a legal search engine that complies with the Cybersecurity Law and the robots conventions.

### How is the DIYSearchEngine system deployed?

As one binary running in four roles, ./ese serve, scheduler, crawler and indexer, with MySQL 8.4 and Redis 7.2 behind them. The compose file shares one environment block across the four services, holding the tuning values such as CRAWL_WORKERS at 64 and CRAWL_BATCH at 4, and the Docker build installs g++ because gojieba needs its CGO part compiled.

## Sources

- [Issues](https://github.com/johnlui/DIYSearchEngine/issues)
- [johnlui/DIYSearchEngine on GitHub](https://github.com/johnlui/DIYSearchEngine)
- [Project website](https://pphc.lvwenhan.com/tech-epic/2023/diy-search-engine)
- [README](https://github.com/johnlui/DIYSearchEngine/blob/main/README.md)
- [Releases](https://github.com/johnlui/DIYSearchEngine/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/johnlui-diysearchengine