SkywalkerDarren/chatWeb: crawl a page, embed it, and ask questions about it

What chatWeb is actually for

The README describes the target case plainly: give it a link or a file path, and it will crawl the page or extract the text, then answer questions about that content or summarize it. The supported inputs are web pages, PDF, DOCX and TXT. That is a narrow, single-source workflow. You are not building a corpus of thousands of documents; you are pointing the tool at one article, one report, one contract, and interrogating it.

The intended user is someone who already has an OpenAI API key and is willing to spend a few cents per document. The README example shows the cost accounting directly: after ingesting an HTML edition of The Old Man and the Sea it reports "Embeddings have been created with 663 embeddings, using 31212 tokens, costing $0.0124848". Embedding a long document is cheap; the query side is where the per-question token count appears, with individual queries in the example consuming around 7,000 tokens each.

This is not a knowledge base product. There is no user model, no sharing, no permissions, no scheduled refresh. If your requirement is "our support team should be able to search our documentation," chatWeb is the wrong shape, because nothing in the repository addresses multi-tenancy or document lifecycle.

The retrieval pipeline, and why keywords are embedded instead of the question

The README lays out the pipeline in order: crawl the page, extract the text, generate embeddings for each paragraph through the OpenAI embedding API, store the vector-to-text mapping, then at query time generate keywords from the user's input, embed those keywords, run a nearest-neighbor search, and pass the most similar texts into a chat completion prompt. The stated goal is to answer from a relevant subset rather than the whole document, which the README frames as a way to work around token limits.

The one design decision worth pausing on is the keyword step. The README says an improvement was made to generate vectors from keywords rather than from the user's raw question, and claims this "increases the accuracy of searching for relevant texts." That is a real trade-off, not a free win. Rewriting a question into keywords can strip conversational noise and pull the query vector closer to the vocabulary of the source paragraphs, which helps when the user asks something wordy. It also means retrieval quality now depends on an extra model call that can drop a qualifier the user cared about. A question like "what did the author say about the 1998 amendment, not the 2004 one" can lose its negation in keyword extraction. The README does not describe how keywords are produced or how many are used, so if retrieval misses, that step is the first place to look.

There is also a summary path in the same pipeline: the README says it calculates a similarity score between each paragraph's vector and the vector of the entire text to produce a summary. That is extractive ranking, not abstractive summarization, so expect selected passages rather than a written digest.

Installing chatWeb and running a first document through it

The manual path is four shell commands plus one config edit. The README gives the clone URL, the directory, the dependency install and the entry point.

git clone https://github.com/SkywalkerDarren/chatWeb.git
cd chatWeb
pip3 install -r requirements.txt
python3 main.py

Before python3 main.py works you need the config file in place. The README says to copy config.example.json to config.json and set open_ai_key to your OpenAI API key. Mode selection is another key in the same file: set mode to console, api, or webui. In console mode the README says to type /help to view commands. In webui mode you set webui_port in config.json, which the README says defaults to http://127.0.0.1:7860.

If you would rather not manage Python locally, the Docker path is documented as three steps. The compose file maps ports 9531 and 7860 and reads OPENAI_API_KEY from the environment, so the key can come from your shell instead of the config file.

docker-compose build
cp config.example.json config.json
docker-compose up

After docker-compose up, the README says to open http://localhost:7860 in a browser. That is the webui mode default, and the same port appears in the compose file and in the Dockerfile EXPOSE lines. In console mode the interaction is a prompt: you paste a URL or a file path, wait for extraction and embedding, and then type questions. The README example shows the prompt text as "Please enter the link to the article or the file path of the PDF/TXT/DOCX document:" followed by a URL, and then the instruction to "wait for 10 seconds until the webpage finishes loading." That is a hint that crawling is time-based rather than event-based, and a slow page can produce a short extraction.

Two more config values shape the answers. Streaming is a boolean, use_stream, set to true in config.json. Temperature is a number between 0 and 1, and the README notes that smaller values give more conservative and stable responses while larger values may produce hallucinations. Proxy support is also configured in config.json under open_ai_proxy, with http and https keys.

FAISS by default, PostgreSQL with pgvector when you opt in

Storage is the part of chatWeb with the clearest fork in the road. The TODO list marks in-memory storage with FAISS as done, and the repository has a storage.py alongside ai.py, contents.py, console.py, webui.py and api.py. FAISS is in requirements.txt as faiss-cpu, so the default path needs no external service.

PostgreSQL is opt-in. The README says to set use_postgres to true in config.json, install PostgreSQL, install the pgvector plugin, and add psycopg2. The default SQL address is postgresql://localhost:5432/mydb, and the README says it can be set in config.json instead. The pgvector extension is pinned to v0.4.0 in the documented build commands, which the README says support Postgres 11+.

git clone --branch v0.4.0 https://github.com/pgvector/pgvector.git
cd pgvector
make
make install

After installing the extension binaries you enable it per database. The README shows the statement as CREATE EXTENSION vector; and then pip3 install psycopg2 for the Python driver. Note that psycopg2 is already listed in requirements.txt, so the pip step is redundant if you installed from that file, though the README presents it as part of the PostgreSQL setup.

The practical difference: FAISS keeps everything in process, which is fine for one document and one user, and means the index lives and dies with the run. pgvector gives you a durable table you can query from other tools and reuse across restarts. The README does not document a migration path between the two, so treat the choice as a decision you make before ingesting, not after.

Where chatWeb breaks down

The clearest limitation is that the README documents no incremental update. Ingestion is described as a single pass over one source, and the example shows the full cost of embedding 663 fragments before you can ask anything. There is no described mechanism for adding a second document to an existing index, removing a stale one, or re-embedding only the paragraphs that changed. If your source is a living document, every refresh is a full re-embed and a full re-payment.

The crawler is the second weak point. The README's own instruction to wait ten seconds before the page is considered loaded suggests the extraction is not waiting on a network-idle signal. The dependency list includes selenium and newspaper3k, and the topics list news-extractor, so the intent is article-shaped pages. A JavaScript-heavy application, a login wall, or a paginated report is outside what the README claims to handle, and nothing in the repository describes authentication for the crawler.

The third limitation is operational. There are no retrieved releases, so upgrades mean pulling master and re-reading config.example.json for new keys. Your config.json is a local file that the Docker setup mounts as a volume, which means a new required key will not appear automatically and the application can fail at startup with no migration note. The README does not document rollback, version pinning, or a changelog.

Finally, cost is per query, not per document. The README example shows individual queries consuming thousands of tokens, with lines like "Query fragments used tokens: 7219, cost: $0.0028876". A chatty session over a long document is the expensive part, and there is no documented cap or budget guard.

How it differs from a general document chat stack

The obvious comparison is a framework that treats loading, splitting and retrieval as separate components, or a hosted notebook like the chatPDF pattern the README itself cites. The difference is scope, not capability. chatWeb is a fixed pipeline: crawl or read, split, embed, store, retrieve by keyword vector, answer. A framework gives you the same steps as interchangeable parts, so you can swap the splitter, the retriever, or the vector store, and add re-ranking or metadata filters.

That flexibility is also why chatWeb exists as a separate thing. If you want to read one article and ask it three questions tonight, editing a JSON file and running python3 main.py is a shorter path than wiring up a chain. The keyword-embedding step is chatWeb's own contribution and is not a standard retrieval pattern, so you will not get it for free by copying a tutorial.

The honest framing: chatWeb trades configurability for a short path to a working answer. If you outgrow it, the exit is not a config change. You would keep config.json's shape as a reference and rebuild the pipeline elsewhere, because storage.py and ai.py encode the retrieval strategy directly rather than exposing it as swappable components.

Licence, maintenance and what an upgrade costs you

The repository is MIT licensed, which permits commercial use, modification and redistribution provided the copyright notice and permission notice are retained. That is permissive and imposes no copyleft obligation on your own code. It says nothing about the OpenAI API terms, which govern your use of the model endpoints the tool calls, and nothing about the licences of the dependencies in requirements.txt. Those are separate obligations you should check against your own policy; this is a description of the licence file, not legal advice.

The repository is not archived, and the last push was on 2026-05-25. There are no retrieved releases, so there is no tagged version to pin to and no changelog to read before upgrading. In practice that means an upgrade is a git pull against master, followed by a diff of config.example.json against your config.json to catch new keys, and a re-run of pip3 install -r requirements.txt in case the dependency set moved. The Docker path makes this slightly safer because the image is rebuilt from the Dockerfile, but the mounted config.json is still yours to reconcile.

The absence of releases also means there is no documented rollback. If a pull breaks your workflow, the recovery is a git checkout of the previous commit, and you should record that commit hash before you pull.

Editorial conclusion

Adopt chatWeb if you want a small, readable Python codebase that turns one URL or one document into a queryable index without standing up a service, and you are comfortable paying OpenAI per embedding and per question. Do not adopt it if you need multi-user access control, incremental re-indexing of a changing corpus, or a vector store other than FAISS and pgvector. Before you commit, verify two things in the code rather than the README: what storage.py does when the process exits in FAISS mode, and whether config.json is read once at startup or on every request.

Frequently asked questions

Official sources

1. Issues
2. License: MIT
3. README
4. SkywalkerDarren/chatWeb on GitHub