# xianyu_spider: a FastAPI search service over Xianyu listings

> xianyu_spider wraps Xianyu's web search into a FastAPI service that fetches listings asynchronously, stores them in a relational database and de-duplicates them by a link feature hash. It also carries a QR code login flow and a private messaging bridge built on a websocket plus server sent events.

**superboyyy/xianyu_spider** — 闲鱼爬虫，基于闲鱼网页端开发，使用 FastAPI 构建的闲鱼商品搜索接口，支持异步并发请求和自动化数据去重存储。现已支持登录。

- Repository: https://github.com/superboyyy/xianyu_spider
- Stars: 890 · Forks: 125
- Language: Python
- License: not declared
- Published: 2026-10-20 · Updated: 2026-10-20 · Language: en
- Canonical page: https://hysenlabs.com/projects/superboyyy-xianyu-spider

## A search API over the marketplace web front end

The project describes itself as a Xianyu product search API built with FastAPI, supporting asynchronous concurrent requests and automated de-duplication and storage of results, and it now supports login. The scope is broader than a search scraper. Alongside keyword search with paging, sorting, price and region filters, it supports an asynchronous high performance crawl that talks directly to the search interface over HTTP and sorts by newest by default. There is QR code scanning login and face verification authentication, with search continuing as a logged out request once the session expires. After login it connects to Xianyu private messages, sending and receiving text, keeping local conversation history and streaming events. On top of that sits data de-duplication based on a hash of link features, and persistence into a relational database, with the response carrying statistics on how many records were new and whether the session is currently logged in.

## The stack and why each piece is there

The technology table lists six components and their jobs. FastAPI is the RESTful API framework, httpx handles the asynchronous HTTP calls for search and login, Playwright is used for face verification, Tortoise ORM is the asynchronous database ORM, SQL is the persistence layer, and Uvicorn is the ASGI server. The pairing of FastAPI with Tortoise and httpx is what makes the whole thing asynchronous end to end, so a search request can fan out over several pages without blocking. Playwright is the outlier and it exists for one reason: the login flow can require face verification, and a headless browser is the only way to complete that. The dependency file pins the library versions with lower bounds, including fastapi 0.68.0, uvicorn 0.15.0, tortoise-orm 0.19.0, playwright 1.32.0, httpx 0.28.1, websockets 12.0, pydantic 2.11.7, aiomysql, aiosqlite, qrcode and Pillow. The comments in that file note that the Chromium browser download has to be run once after installing the package, and that the MySQL driver should be swapped for asyncpg if you use PostgreSQL.

## Installing and starting the service

The quick start is three steps. Install the dependencies, and install the Chromium build that the face verification flow needs, which the instructions say only has to be done once.

```bash
pip install -r requirements.txt
playwright install chromium
```

Next, create an .env file. That step is optional, because without it the service falls back to a SQLite file at data/xianyu.sqlite3. If you do set one, the only variable shown is the database URL, and it takes a MySQL style connection string.

```bash
DATABASE_URL=mysql://user:password@localhost/xianyu
```

Then start the server, which runs the spider module as a script.

```bash
python spider.py
```

The interactive API documentation is served at the docs path on port 8000, so once the process is up you can read the request schemas directly in the browser instead of guessing field names.

## Searches take filters and report what is new

The search endpoint is a POST to /search/ and the request body carries the keyword, a page count, a sort order and optional price and city bounds. Four sort values are documented: newest as the default, meaning most recently published, price_asc, price_desc and default, the general ordering. The response echoes the keyword, the filters that were applied, the total number of results, and two fields that make the de-duplication useful: new_records, a count, and new_record_ids, the identifiers of the listings that were not already stored. The CLI offers the same search with the same options, and a no-save flag that prints the JSON without writing anything to the database. A Python client against a running service is a plain POST to the same path with a keyword and a page count in the JSON body.

## Login by QR code, by pasted cookie or by the official page

The login command prints a QR code in the terminal, and once you scan and confirm it, a browser opens automatically if face verification is required. Two alternatives exist for other situations.

```bash
python spider.py login --cookie
python spider.py login --browser
```

The cookie variant pastes in the cookie of an already logged in browser session, and the browser variant opens the official login page directly. The session state is written to data/session.json, and a GET to /auth/status asks Xianyu whether the stored cookie is still valid, reporting the logged out state when it is not. That expiry is handled rather than treated as fatal. When the cookie has expired, search continues as a logged out request and returns HTTP 200 with logged_in set to false and a login_expired marker in the payload. Endpoints that genuinely need a session return 401 instead.

## Private messaging over a websocket with an SSE feed

The messaging side needs a running service plus an open Xianyu IM long connection, and it exposes a small set of endpoints. The start endpoint opens the connection, the status endpoint reports on it, the conversations endpoint lists conversations, and the messages endpoint fetches history for a given conversation id. Sending is a POST with a conversation id, a recipient id and the text, and the events endpoint is a server sent events stream that emits message.received and message.sent notifications.

```bash
curl -X POST http://127.0.0.1:8000/im/start
curl http://127.0.0.1:8000/im/status
curl http://127.0.0.1:8000/im/conversations
```

The status endpoint carries three counters, and the distinction between them is the useful part: ws_frames, sync_pushes and parsed. A parsed value of zero with sync_pushes above zero means frames arrived and the decoder could not read them, while ws_frames at zero means the connection came up and the server pushed nothing. Two operational warnings are documented. The history comes from the local database and only covers text sent or received after the process connected, so it is not a complete archive. And the official web IM must not run at the same time, because the token will evict one of them.

## De-duplication turns repeated searches into new item counts

The distinguishing feature of the project is that a search is not merely a query result. The crawler computes a hash of link features for each listing, uses that to recognise an item it has already stored, and inserts only the unseen ones. The response then tells you how many new records that search produced and which identifiers they were. In practice this turns a polling loop into a change feed: run the same keyword repeatedly and the count drops to zero once the market has been harvested. The repository root shows the shape of the code, with api.py holding the FastAPI routes, spider.py the command line entry point, an xianyu package for the internals, plus test.py and a tests directory and a pytest.ini. The project also ships a test client example using the requests library against the running service.

## Rate limits, the platform's terms and the licence question

The project's own notes are the part to read carefully. On legal compliance, it asks users to follow the Chinese cybersecurity law and the Xianyu platform's robots agreement, and states that the code is for study and research. On anti-crawler measures, it advises setting up a proxy IP pool and randomised request intervals because the default configuration may trigger rate limits. On performance tuning, it suggests adjusting the database connection pool parameters such as pool_recycle and adding a Redis caching layer for production deployments. Those three notes are not incidental; they describe a service that will get blocked if you run it hard. On licensing, the copyright statement says the project uses the MIT License and asks for reasonable use with attribution, and adds that scraped data may not be used commercially, while the licence field on the repository itself carries no standard identifier. The repository has no GitHub releases, and the last push was on 2026-09-18.

## Conclusion

Use xianyu_spider when you want Xianyu search results as JSON from your own code rather than by scraping HTML yourself, since the service handles paging, sorting, filters, de-duplication and persistence behind a single POST /search/ endpoint. Do not build it into anything that depends on stable behaviour or a clear licence, because the repository has no GitHub releases, the licence field on the repository carries no standard identifier even though the copyright notice names the MIT License, and the README asks for proxy pools and randomised intervals to avoid triggering rate limits. Verify first that your platform's terms and the robots rules permit what you are about to request, that the database URL in your .env points at a reachable instance, and that the endpoints you call return the login_expired flag before you assume an empty result means no listings exist.

## FAQ

### How do I install and run xianyu_spider?

Install the dependencies with pip install -r requirements.txt and then run playwright install chromium once for the face verification flow. Start the service with python spider.py and the interactive API documentation is served on port 8000 under /docs.

### How do I log in to xianyu_spider?

Run python spider.py login, which prints a QR code in the terminal, and scan it. If face verification is required a browser opens automatically. You can also paste an existing cookie with python spider.py login --cookie or open the official login page with python spider.py login --browser.

### What happens when the login session expires?

The GET /auth/status endpoint asks Xianyu whether the cookie is still valid and reports the logged out state when it is not. Search then continues as a logged out request and returns HTTP 200 with logged_in set to false and a login_expired marker, while endpoints that need a session return 401.

### How does xianyu_spider avoid storing the same listing twice?

It computes a hash of link features for each listing and uses it to recognise items already stored, inserting only unseen ones. The search response reports how many records were new and lists their identifiers.

### Which database does xianyu_spider use?

By default it stores data in a SQLite file at data/xianyu.sqlite3, and the .env file is optional. You can point it at MySQL or PostgreSQL by setting DATABASE_URL, with the driver noted in the requirements file.

## Sources

- [Issues](https://github.com/superboyyy/xianyu_spider/issues)
- [README](https://github.com/superboyyy/xianyu_spider/blob/main/README.md)
- [superboyyy/xianyu_spider on GitHub](https://github.com/superboyyy/xianyu_spider)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/superboyyy-xianyu-spider