UAJY Academic RAG Chatbot: a grounded handbook assistant built on FAISS, BM25 and an OpenAI-compatible gateway

What the UAJY academic RAG chatbot is actually for

The README frames the problem narrowly and that is its strength. Students and faculty ask administrative questions about the Buku Pedoman Akademik Fakultas Teknologi Industri Universitas Atma Jaya Yogyakarta 2025/2026: grade thresholds, credit requirements, article numbers. A general model answers those questions fluently and sometimes wrongly, which is worse than refusing. The project splits the job in two. Retrieval runs locally against FAISS and BM25; generation runs against an OpenAI-compatible gateway. Answers are supposed to come only from retrieved PDF chunks, and out-of-scope questions are supposed to be refused rather than answered.

So the intended user is not a chatbot hobbyist. It is a department that already has a canonical PDF and wants an interface over it that cites pages. The repository ships a Streamlit UI described as SIATMA-inspired, a document explorer over the index, and an evaluation harness. That is a lot of surface for a single-document assistant, and it tells you the author cares about being able to prove the thing works rather than about breadth of coverage.

How hybrid retrieval, RRF and the reranker gate fit together

The pipeline in the README is explicit about ordering. A user query arrives through Streamlit. Before any retrieval happens, a history-aware rewriting step turns a follow-up such as "berapa maksimalnya?" into a standalone question, because a fragment retrieved on its own will not match anything useful. Then two retrievers run in parallel: FAISS cosine similarity with top-20, and BM25 Okapi with top-20. The two rankings are merged with Reciprocal Rank Fusion, which the README notes is rank-based and needs no score normalization. That choice matters here, because cosine distances and BM25 scores are not on a comparable scale and normalizing them is where hybrid systems usually go wrong.

After fusion, a listwise reranker scores candidates on whether they can answer the question rather than whether they resemble it. The README reports the measured effect as MRR 0.858 to 1.000. Then guardrails run in layers: a similarity floor, IDF-weighted lexical coverage, the reranker gate, and a strict system prompt. The README states out-of-scope refusal is 100% in the full configuration. Treat that number as a property of this handbook and this evaluation set, not a general guarantee.

Two ingestion details change how the system behaves in practice. Chunks are built with per-line page tracking, and the README says 76% of chunks cite a single page, averaging 1.29 pages per chunk. And 24% of raw lines are filtered at ingestion, because rotated org-chart diagrams and broken font encodings produce text that would otherwise compete for top-k slots. Noise filtering is doing real work in the ranking, not just cleaning output.

Installing it and running a first query

The repository declares Python 3.10+ and lists its runtime dependencies in requirements.txt, including streamlit==1.49.1, faiss-cpu, PyMuPDF and google-genai. Create a virtual environment and install them:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

A development install adds the test dependencies. The README states the suite covers every pure-logic module and runs without an API key, so this is the cheapest way to confirm the environment is sane before you touch credentials:

pip install -r requirements-dev.txt
pytest

Ingestion is described as offline and one-time: PDF text extraction with pdfplumber, noise filtering, page-accurate chunking with heading paths, then embeddings, producing faiss.index, metadata.json and index_info. The README does not print the exact ingestion command, so read ingestion/ for the entry point rather than guessing a flag. The same applies to launching the UI; app/ holds the Streamlit code and .streamlit/ holds its configuration, but no run command appears in the README.

What you should expect after a successful build: an index spanning the indexed chunks, a document explorer that lets you search and filter them with their hierarchical section paths, and a per-document search filter so that "page 48" resolves against the right source. The README gives 218 indexed chunks as the figure for the current index.

Where the design breaks down

The embeddings come from Gemini, and the README labels that arrangement temporary. The gateway handles chat, rerank and rewrite in what it calls the current short-term deployment. Two external services with a stated expiry on one of them is the first thing to plan around: if the embedding model changes, the FAISS index has to be rebuilt, because vectors from different models are not interchangeable. There is no migration path in the README.

The second constraint is scale. The index holds 218 chunks from one handbook. Hybrid retrieval with a listwise reranker is affordable at that size and stops being so as the corpus grows, since the reranker scores every candidate. The multi-document support and the per-document filter suggest the author intends to add more sources, but nothing in the README describes how fusion or reranking behaves when the candidate pool spans dozens of documents with overlapping vocabulary.

Third, the guardrails trade recall for precision. A similarity floor plus lexical coverage plus a reranker gate will refuse questions that are answerable but phrased in vocabulary the handbook does not use. The project measures false-refusal rate in its ablation study, which is the right instinct, but the README does not state an acceptable ceiling for it. For a student asking about a deadline, a refusal is a support ticket.

Finally, staleness detection is a warning, not a repair. SHA-256 and a pipeline version are recorded at build time and re-checked on startup, so a changed PDF or improved chunking surfaces as a warning. Nothing documented re-ingests automatically, and the README does not document rollback.

Compared with a hosted document assistant

The obvious alternative is a managed retrieval product or a hosted assistant over the same PDF, where you upload the document and get an endpoint. The difference is where the ranking logic lives. A hosted product chooses the chunking, the retrieval mode and the refusal behaviour, and exposes at most a handful of knobs. This project puts the whole path in the repository: you can read the noise filter, change the fusion method, inspect the reranker prompt, and rerun the ablation study to see what your change did. That is the reason to pick it.

The reason not to is operational. A hosted assistant does not need a FAISS index rebuilt when an embedding model is deprecated, and it does not need someone to notice a startup warning about a changed SHA-256. If your team has no Python maintainer and no appetite for re-indexing, the repository's control is a liability rather than a feature. Between those two positions there is a middle option worth naming: run the same hybrid retrieval but swap the listwise reranker for a cross-encoder, which removes one API dependency at the cost of the listwise scoring the README reports as the source of its MRR gain. The project does not implement that, and the ablation harness is where you would measure whether it is worth it.

Maintenance cost, licensing and what to check before adopting

The repository is MIT licensed, which permits commercial and academic reuse with the usual requirement to keep the copyright and permission notice. That is the extent of what the licence file settles. It says nothing about the terms of the gateway or the embedding provider, and those are separate agreements you would need to read yourself; the README does not discuss data retention or what the gateway does with submitted queries, which matters if student questions are routed through it.

Maintenance-wise, the last push was on 2026-09-16, so the code is current as of this writing. There are no retrieved releases, which means you would track the main branch rather than pin a version. The dependency list is small and mostly stable, with two exceptions worth noting: streamlit is pinned exactly at 1.49.1, and google-genai is a fast-moving SDK. An exact Streamlit pin protects you from UI regressions but also blocks security patches until someone bumps it deliberately.

The upgrade path that will actually cost you time is re-ingestion, not code. Any change to the embedding model, the noise filter or the chunking logic invalidates the index and requires a rebuild plus a rerun of the ablation study to confirm the metrics still hold. Budget for that as a recurring task rather than a one-off, and check the ingestion entry point in ingestion/ before you assume it is a single command.

Editorial conclusion

Adopt it if you need a worked example of hybrid retrieval, rank fusion and a reranker gate on a small, page-addressable document set, and you are willing to supply your own PDF and embedding credentials. Do not adopt it as a drop-in campus assistant for a different university: the index, the prompt and the refusal thresholds are tuned to one handbook, and the README does not document rollback or re-ingestion procedures. Verify first that the extraction pipeline handles your PDF, since rotated org charts and broken font encodings are filtered at ingestion and 24% of raw lines are dropped; then confirm which service is actually serving chat, because the README describes Gemini embeddings as temporary and the gateway as the current short-term deployment.

Frequently asked questions

Official sources

1. BenyRonald77/uajy-academic-rag-chatbot on GitHub
2. Issues
3. License: MIT
4. README