UAJY Academic RAG Chatbot: Hybrid FAISS and BM25 Retrieval Over One University Handbook
Production-grade RAG chatbot for Universitas Atma Jaya Yogyakarta academic handbook with Streamlit, FAISS vector search, and Google Gemini 2.5 Flash.
At a glance
- What is it?
- A Streamlit question-answering app for the Universitas Atma Jaya Yogyakarta academic handbook. It pairs local FAISS and BM25 retrieval with a gateway-hosted LLM, and its own README says the current provider setup has not been re-evaluated.
- Who is it for?
- Use it as a worked example of hybrid retrieval, rank fusion and stale-index checks on a single Indonesian university handbook. Do not treat its accuracy numbers as current: the README says retrieval and answer evaluations must be rerun on the present provider setup, and the index still needs a rebuild to fix heading noise.
- Can I use it commercially?
- Yes. MIT is a permissive licence: you can use, modify and sell software built on it, as long as you keep its copyright and licence notices.
- Is it still maintained?
- Yes. The repository last received commits 59 days ago.
- What is it written in?
- Mainly Python, according to GitHub's language statistics.
Answers come from the project's GitHub data, last synced on November 15, 2026, and from our analysis. They are not legal advice.
Star history
- BenyRonald77/uajy-academic-rag-chatbot453 ★ · -187
| Repository | 2026-09-17 | 2026-11-15 |
|---|---|---|
| BenyRonald77/uajy-academic-rag-chatbot | 640 | 453 |
-187 Stars in the last 59 days, measured from the daily GitHub snapshot. Compare with other projects
Editorial analysis
Retrieval stays on your machine while only the wording of the answer leaves it
The project splits two jobs that many chatbots blur together. Finding the right passage is done locally with a FAISS vector index and a BM25 Okapi keyword index. Writing the reply is done by an OpenAI-compatible LLM gateway. The README frames this as the reason the bot can cite the official handbook, Buku Pedoman Akademik Fakultas Teknologi Industri Universitas Atma Jaya Yogyakarta 2025/2026, instead of improvising administrative deadlines or degree requirements.
The practical consequence is that the knowledge base is whatever PDFs were ingested, and nothing else. Answers are meant to be generated exclusively from retrieved chunks, and out-of-scope questions are meant to be refused. A student asking about a different faculty, or about a rule that is not in the ingested documents, should get a refusal, not a guess. The flip side is that the bot knows nothing about the world outside its index, so it cannot help with questions the handbook does not cover.
The repository description still names Google Gemini 2.5 Flash as the generator, but the README describes a changed split: the gateway now handles chat, reranking and query rewriting, and Gemini is used only for embeddings. Read the README, not the one-line description, when you plan what to configure.
Rank fusion is used because cosine scores and BM25 scores share no scale
Academic documents are full of tokens that a dense embedding blurs: `IPK 3,51`, `144 SKS`, `Pasal 12`. The README gives an example of the failure. Ask "berapa IPK minimum untuk cum laude?" against a dense-only index and it happily returns chunks about graduation in general. BM25 anchors on the literal terms, so the project runs both retrievers.
Merging them is the delicate part. Cosine similarity lives between 0 and 1 while BM25 is unbounded, and normalising one against the other needs arbitrary scaling. The project uses Reciprocal Rank Fusion, which sums `weight / (k + rank)` so that only the ordering of each list matters, not the raw score.
What this does not give you is any notion of how good the best hit is. Because fusion discards absolute scores, a question with no real answer in the handbook still produces a ranked list. The project therefore stacks further checks on top: a similarity floor, an IDF-weighted lexical coverage test, a reranker gate and a strict system prompt. If you adapt the retrieval code to your own documents, you inherit the need to tune those gates, since rank fusion alone will never say "nothing relevant was found".
The reranker is a single gateway call because a local cross-encoder would drag in PyTorch
After fusion, a listwise LLM reranker scores every candidate on whether it can actually answer the question, not merely resemble it. The README explains why it chose an LLM call over a local multilingual cross-encoder: those models weigh hundreds of megabytes and pull in PyTorch, which conflicts with keeping the project CPU-only and deployable on free tiers. One provider-gateway call scores all candidates instead.
That trade has a cost that the README accepts rather than hides. Every question now depends on a network round trip to a third-party gateway before an answer can be shown, and the same gateway also rewrites follow-up questions into standalone ones before retrieval runs. A query like "berapa maksimalnya?" is turned into a full question first, so the right context is fetched on the first pass. If the gateway is down, the reranking and rewriting steps go with it.
For a deployment that must work offline or on a locked-down network, this design is the wrong fit. The local pieces (FAISS, BM25, PDF parsing) will run, but the stage that decides which chunks the model sees does not. The requirements file reflects the CPU-only goal, listing streamlit, pandas, PyMuPDF, pdfplumber, faiss-cpu, google-genai and numpy, with no torch in sight.
Two providers and two keys: the gateway has no embeddings endpoint and auto returns 403
The provider setup is the most fragile part, and the README is specific about it as of 17 September 2026. The gateway at `https://bandelbanget.xyz/v1` handles chat completions, reranking and query rewriting. You must name the model explicitly as `deepseek-v4-flash`, because `auto` currently returns `403 model_disabled`. When a model-specific disabled or out-of-stock error comes back, the app retries with `glm-5.3-flash`. Both models passed connection and JSON-mode checks, and DeepSeek passed a grounded-answer smoke test after the prompt was changed to include copyable source lines.
The gateway does not expose `/embeddings`. Embeddings therefore come from Google's `gemini-embedding-001` at 3072 dimensions, and the README tells you to keep the Bandel and Google keys separate. In practice you need two accounts and two secrets before the first question works.
The lock-in runs deeper than the keys. The index is built with Gemini vectors, so swapping the embedding provider means rebuilding the whole index, not changing a setting. And because the chat model names are tied to what one gateway currently enables, a model that is disabled upstream breaks the explicit name you configured until the fallback kicks in. Cost depends entirely on those providers' own pricing, which the repository does not set.
The published accuracy figures describe an older provider setup
The README opens with a validation status dated 17 September 2026. Verified: 456 offline tests pass, public and operator modes work, rate limiting and the answer guard work, page citation parsing is precise, and the PDF viewer works, with the native PDF viewer button in commit `8ef96ac`. Still pending: rerunning retrieval and answer evaluations with the current provider setup, correcting ALL-CAPS heading noise and rebuilding the index, a browser-based end-to-end UI check, and VPS deployment.
The sentence that matters most is that the evaluation tables are historical results and do not describe the measured performance of the current provider setup. The reranker's MRR improvement and the refusal rate are both flagged the same way: they come from an earlier evaluation and need remeasuring.
For a reader, that means two things. First, you cannot cite this project's retrieval quality or refusal accuracy for the DeepSeek plus Gemini configuration, because nobody has measured it yet. Second, the ablation study that compares dense-only, hybrid and hybrid with rerank on page-level relevance, MRR, refusal accuracy and false-refusal rate is a method you can reuse, but its tables are not evidence about the code as it runs today. The 456 passing tests cover pure-logic modules and regression guards, which is a different thing from answer quality.
A stale index raises a warning instead of silently serving old answers
Each document's SHA-256 and an ingestion pipeline version are recorded when the index is built, then re-checked on startup. Two situations trip the check: a source PDF that changed, and chunking logic that improved. Both surface as a warning, so the app does not quietly answer from an index that no longer matches its sources.
The check is a detector, not a fixer. It tells you the index is out of date but does not rebuild it, so someone still has to rerun ingestion. The README's own pending list shows why this matters right now: ALL-CAPS heading noise is known and needs a rebuilt index to correct. Until that happens, the hierarchical section paths shown in the document explorer carry that noise.
The index is also multi-document. One index spans many campus documents and every chunk records its source, because page 48 means something different in the academic handbook than in a rector's decree. A per-document search filter is included. The explorer lets you search, filter and inspect all 218 indexed chunks with their section paths, a useful size figure: this is a small corpus, not a university-wide search engine, and nothing in the README suggests it was tuned for large collections.
Noise filtering drops 24% of raw lines and still leaves citations at 1.29 pages per chunk
Rotated org-chart diagrams and broken font encodings in the PDF produce garbage text that would otherwise compete for top-k slots. Ingestion extracts text with pdfplumber and filters about 24% of raw lines as noise. That is an aggressive cut, and it is applied to the handbook as extracted; a differently laid-out PDF may lose real content to the same filters or keep noise the filters do not know about.
Chunks are built with per-line page tracking, which is how the bot cites pages. The README reports that 76% of chunks cite a single page, with an average of 1.29 pages per chunk. Read the other side of that figure: roughly a quarter of chunks span more than one page, so a citation for those points at a range, not at the exact page where a rule appears. A student checking a graduation requirement may need to look at neighbouring pages.
The interface is a Streamlit app with a light blue page background, white cards and cyan section headers inspired by SIATMA, plus a public student mode. The project is MIT licensed and has no GitHub releases, so there is no tagged version to pin. The last push was on 2026-09-17, and the code in the repository is the only distribution.
Editorial conclusion
Use it as a worked example of hybrid retrieval, rank fusion and stale-index checks on a single Indonesian university handbook. Do not treat its accuracy numbers as current: the README says retrieval and answer evaluations must be rerun on the present provider setup, and the index still needs a rebuild to fix heading noise.
Frequently asked questions
What does RAG chatbot mean in UAJY Academic RAG Chatbot?
RAG stands for Retrieval-Augmented Generation. In this project, local FAISS and BM25 search find handbook passages, and an LLM gateway writes the answer from those passages only.
How much does the UAJY Academic RAG Chatbot cost to run?
The code is MIT licensed. Running it needs a Bandel AI gateway key for chat and reranking and a Google key for Gemini embeddings, so the cost is whatever those two providers charge.
Which models does the UAJY Academic RAG Chatbot use?
The gateway uses deepseek-v4-flash for chat, rewriting and reranking, with glm-5.3-flash as a fallback. Gemini's gemini-embedding-001 provides 3072-dimensional embeddings.
Are the UAJY Academic RAG Chatbot accuracy numbers current?
No. The README says the evaluation tables are historical and that retrieval and answer evaluations must be rerun with the current provider setup.
Official sources
Add this badge to your README
If you maintain this project, the badge below links readers to this analysis and shows its maintenance status from the daily GitHub snapshot. Paste the markdown into your README; add ?metric=license or ?metric=stars to the image URL for a different field.
[](https://hysenlabs.com/projects/benyronald77-uajy-academic-rag-chatbot)Star history chart for your README
[](https://hysenlabs.com/projects/benyronald77-uajy-academic-rag-chatbot)Was this useful?