finic-ai/rag-stack: Private RAG Chat with GPT4All, Falcon, and Qdrant

Why finic-ai/rag-stack Exists for Private Document Chat

finic-ai/rag-stack addresses the problem of asking an LLM questions about an organization's documents without sending that workflow to a public chatbot service. The intended audience is a company or technical team that needs a chat interface over internal material such as PDFs. The README identifies sales contracts, PDFs, Confluence, and Salesforce as examples of enterprise information, although the shipped UI is specifically documented as handling PDF uploads.

The project uses retrieval-augmented generation, or RAG. In this design, information is retrieved from another system and inserted into the LLM's prompt as context. That distinction matters because the model does not have to rely only on knowledge encoded during training. It can receive current project documents as part of answering a question.

The repository also places the server, model, vector database, and user interface inside infrastructure controlled by the deployer. The README calls the result a private ChatGPT alternative hosted within a VPC, but it does not document every control needed to establish that property. In particular, the README does not describe network restrictions, encryption configuration, audit logging, retention policies, or authentication tests. The software supports a private deployment model, but those operational controls still need separate verification.

The repository's primary language is TypeScript and its top-level layout includes `ragstack-ui/`, `server/`, `scripts/`, `python/`, and `containerize-llms/`. That split suggests a UI component, server-side retrieval and model integration, deployment automation, and model containerization rather than a single command-line chat tool. The documentation does not expose a public API contract, so callers should not assume that an SDK or stable HTTP interface is part of the supported surface.

The GPT4All, Qdrant, PDF, and LLM Request Path

The documented architecture contains four principal parts: an open-source LLM, Qdrant as the vector database, a server, and a UI for PDF upload and chat. The server handles PDF upload according to the README. The retrieval path depends on Qdrant, which stores and searches vector data in a self-hosted Rust database. Relevant document content can then be included in the prompt sent to the selected LLM.

There are two execution models. For local use, RAGstack downloads Nomic AI's `gpt4all` model, identified in the setup instructions as `ggml-gpt4all-j-v1.3-groovy.bin`, and places it in `server/llm/local/`. The README states that this model runs on consumer CPUs. The local launcher starts the server, LLM, and Qdrant vector database together.

For cloud deployment, the documented default is Falcon-7B on GPU infrastructure. Google Cloud uses a GPU-enabled GKE cluster, AWS uses ECS on EC2, and Azure uses AKS. RAGstack can also deploy the 7B parameter version of Llama 2 to a GPU-enabled GKE cluster. The cloud deployment scripts are described as Terraform implementations.

This design separates model execution from retrieval storage, which gives an adopter some deployment choice, but it also creates several integration points. A successful deployment must coordinate model weights, GPU access where required, Qdrant connectivity, server configuration, and frontend routing. The README does not document chunking policy, embedding model selection, retrieval count, reranking, citation format, or behavior when a PDF contains no relevant text. Those omissions prevent a reliable assessment of answer grounding from the documentation alone.

Running the First Local PDF Chat Session

The local path is the most direct first use because it avoids the GPU-oriented cloud deployment scripts. The README requires two environment files, Supabase credentials, and a `ragstack_users` table before it starts the stack. The server file is `server/.env`, copied from `server/example.env`. The UI file is `ragstack-ui/.env`, copied from `ragstack-ui/local.env`.

In `server/.env`, replace `YOUR_SUPABASE_URL` with the Supabase project URL and `YOUR_SUPABASE_KEY` with the Supabase secret API key. In `ragstack-ui/.env`, replace `YOUR_SUPABASE_URL` with the project URL and `YOUR_SUPABASE_PUBLIC_KEY` with the documented public key value. The README points users to Supabase's API settings for these values. Despite calling one value a secret API key in the server file, users should check the current Supabase key model before treating any stored credential as safe.

Create a table named `ragstack_users` with `id`, `app_id`, and `secret_key` typed as `uuid`, plus `email`, `avatar_url`, and `full_name` typed as `text`. The README does not supply SQL for this table, so no executable DDL is reproduced here. If row-level security is enabled, it requires an insert and select `WITH CHECK` expression of `(auth.uid() = id)`.

After the files and table are ready, run the repository launcher from its root. This is the exact command documented by the README.

scripts/local/run-dev

The launcher downloads `ggml-gpt4all-j-v1.3-groovy.bin` into `server/llm/local/` and starts the server, LLM, and Qdrant. The documented readiness message is:

INFO:     Application startup complete.

The README does not give the local UI URL, a sample PDF, or a sample chat prompt. Those details must be obtained from the running application or source rather than inferred.

Google Cloud, AWS, and Azure Deployment Paths

Cloud deployment moves inference from the local GPT4All arrangement to GPU-backed infrastructure. The README presents Google Cloud as using Falcon-7B on a GPU-enabled GKE cluster, AWS as using ECS on EC2, and Azure as using AKS. Each deployment path is controlled by a script that prompts for infrastructure-specific values such as credentials, project identifiers, regions, model selection, or a HuggingFace token.

For Google Cloud, the documented entry point is:

scripts/gcp/deploy-gcp.sh

The script asks for a GCP project ID, service account key file, region, model, HuggingFace token, and other parameters. If the Falcon-7B stage fails, the README gives a recovery sequence. Replace the zone placeholder with the selected region, then apply the NVIDIA driver installer and rerun the deployment script.

gcloud config set compute/zone YOUR-REGION-HERE
gcloud container clusters get-credentials gpu-cluster
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml

The AWS path uses:

scripts/aws/deploy-aws.sh

It requests AWS credentials and other parameters. The Azure path uses:

./azure/deploy-aks.sh

Azure asks for an AKS subscription and other parameters. Its documented accelerator configuration uses an NVIDIA Tesla T4 node pool, but the README warns that this hardware is not available in every subscription.

For either cloud UI, create a `.env` file in `ragstack-ui` and set `VITE_SERVER_URL` to the deployed server URL. The README does not state which URLs are exposed publicly, whether authentication is enabled, or what network rules the Terraform modules create. Those checks must happen before sensitive documents are uploaded.

Maintenance State, Releases, and MIT License Implications

Maintenance status requires careful wording here. The repository is not archived, and its last push was on 2026-10-12. However, the README explicitly labels RAGstack as "a legacy project" and says it "is no longer maintained." That project-specific statement is the stronger warning for an adoption decision, even though recent repository activity and non-archived status might otherwise suggest ongoing work.

GitHub has no releases for the repository. The README therefore offers no formal versioned upgrade path, and it does not document a compatibility policy for GPT4All, Qdrant, Falcon-7B, Llama 2, Supabase, Terraform, GKE, ECS, or AKS. The deployment instructions also reference external models and cloud services that can change independently of this repository.

The repository uses the MIT License. That license permits use, modification, distribution, and sublicensing under its conditions, including preservation of the copyright and license notice. This is a factual license summary, not legal advice. For internal use, an organization should still review dependencies separately because the MIT license for this repository does not automatically settle the terms of linked models, cloud services, or third-party components.

Upgrade cost is difficult to estimate from the documentation. There is no migration guide, release history, supported-version matrix, or rollback procedure. A team taking ownership would need to verify every deployment script, model URL, package dependency, and integration manually.

Why finic-ai/rag-stack Is the Wrong Choice for Some Teams

The clearest wrong-tool case is a new team seeking a maintained production platform with current support expectations. The README says the project is no longer maintained, the repository has no GitHub releases, and the cloud deployment paths depend on several external systems. A team that cannot assign ownership for model downloads, infrastructure provisioning, dependency upgrades, and retrieval quality should not treat the repository as a finished internal platform.

It is also a poor fit for users whose primary requirement is access to broad SaaS knowledge. The README mentions Confluence and Salesforce as RAG examples, but the shipped server and UI are specifically documented for PDF upload. No connector or ingestion workflow for those SaaS products is described.

CPU-only teams can use the local GPT4All route, but teams needing Falcon-7B or Llama 2 should expect GPU infrastructure. The Azure instructions additionally depend on an NVIDIA Tesla T4 node pool that may not be available in the target subscription. Model selection is therefore tied to both workload and cloud access.

The documentation also does not explain how answers cite retrieved content, how documents are deleted, or how one user is prevented from retrieving another user's data. Those omissions are especially important for corporate documents. They do not prove that the system lacks these controls, but they do mean the README cannot support a production security assessment.

RAGstack Compared with a Hosted Managed RAG Service

A real alternative is a managed RAG platform that owns ingestion, retrieval, model serving, upgrades, and operational monitoring while the customer supplies documents through a hosted workflow. The practical difference is ownership. finic-ai/rag-stack lets an engineering team run Qdrant, an open-source model, a server, and a UI inside infrastructure it selects, including GKE, ECS, or AKS. The customer accepts responsibility for Terraform state, GPU provisioning, model compatibility, application security, and ongoing maintenance.

A managed service shifts much of that work to the provider, but it does not necessarily satisfy a requirement for every component to remain inside the customer's VPC. The README does not describe a managed RAG vendor or a contractual data-processing model, so no vendor-specific comparison can be claimed. The choice is architectural rather than a claim that one category always performs better.

RAGstack may fit an organization that already has Kubernetes, cloud operations, and MLOps staff, accepts legacy code, and needs a concrete PDF-chat implementation to adapt. It may not fit a small team seeking current releases, predictable infrastructure support, SaaS connectors, or documented production operations. In particular, RAGstack's MIT-licensed code is easy to inspect and modify, while operational ownership is not solved by an open-source license.

Editorial conclusion

Organizations already evaluating this repository for private document chat should adopt it only as a legacy reference implementation after checking model access, cloud costs, PDF retrieval behavior, and security ownership. It is a poor fit for a new production deployment because the README explicitly says the project is no longer maintained, GitHub has no releases, and no current compatibility or upgrade guarantees are documented. The first adoption check should be whether the pinned GPT4All model and the supplied Google Cloud, AWS, or Azure deployment scripts still run with the intended infrastructure.

Frequently asked questions

Official sources

1. finic-ai/rag-stack on GitHub
2. Issues
3. License: MIT
4. Project website
5. README