ray-project/llm-applications: A RAG Reference Stack You Run Yourself

The problem: RAG demos that never leave the notebook

Most retrieval augmented generation examples stop at a single script that embeds a folder of text and prints an answer. The hard part is everything after that: chunking choices, embedding throughput, index construction, a serving path, and some way to tell whether a configuration change made the answers better or worse. This repository is aimed at that second half. The README frames it as a guide to developing a RAG application from scratch, scaling the major components (load, chunk, embed, index, serve), evaluating configurations against per-component and overall scores, and serving the result. The intended reader is an engineer who already knows what RAG is and wants a reference implementation of the surrounding machinery, not a conceptual introduction. It is a guide, not a library. Nothing here is published to PyPI; you clone the repository and work through notebooks/rag.ipynb. That distinction matters when you decide whether to adopt it, because you are adopting a pipeline structure and a set of patterns rather than a dependency you can pin and upgrade.

What the pipeline actually does, stage by stage

The README names the stages explicitly: load, chunk, embed, index, and serve, plus an evaluation step that scores both retrieval and final answer quality. The evaluation section is the most interesting design choice, because it splits measurement into a per-component score (the README gives retrieval_score as the example) and an overall quality_score. That split lets you change a chunking parameter and see whether retrieval moved, separately from whether the generated answer moved. The serving stage uses Ray Serve, based on the release history: v0.0.7 is titled "Fix serve application to use Anyscale Endpoints" and v0.0.9 is "Update Ray Assistant Service". The routing idea in the README, described as a hybrid routing approach to bridge the gap between OSS and closed LLMs, implies a request path that can send some traffic to an open model and some to a hosted one. The README does not specify the routing rule, the thresholds, or the fallback behaviour, so treat that as something to read out of the notebook rather than assume. The data flow is otherwise conventional: source documents are loaded, split into chunks, embedded, written to a vector store, retrieved at query time, and passed to a model for generation. The repository's contribution is making each of those steps a separately scalable Ray stage.

Getting it running: the commands and keys the README specifies

Setup starts with cloning and environment configuration. The README gives: git clone https://github.com/ray-project/llm-applications.git, then pip install --user -r requirements.txt, export PYTHONPATH=$PYTHONPATH:$PWD, and pre-commit install followed by pre-commit autoupdate. Credentials live in a .env file created with touch .env, and the README lists four keys: OPENAI_API_BASE, OPENAI_API_KEY, ANYSCALE_API_BASE, and ANYSCALE_API_KEY, plus a DB_CONNECTION_STRING whose example value is dbname=postgres user=postgres host=localhost password=postgres. The file is loaded with source .env. Compute guidance is specific to Anyscale: the README suggests an Anyscale workspace with a g3.8xlarge head node (2 GPUs, 32 CPUs), the default_cluster_env_2.6.2_py39 cluster environment, and the us-west-2 region if you want to use the shared storage artifacts. The shared data is described as already present at /efs/shared_storage/goku/docs.ray.io/en/master/ on Staging. That path is an internal detail rather than a general instruction, and anyone outside that environment has to point the loader at their own documents. The README also notes you can run locally but recommends GPU access, which is honest about what the embedding and serving stages cost.

Where it stops being the right tool

The dependency on Anyscale Endpoints is the first constraint. The README directs you to create accounts with both OpenAI and Anyscale Endpoints before you begin, and the release notes show the serve application was fixed to use Anyscale Endpoints specifically. If your organization has standardized on a different inference provider, or on self-hosted weights only, you are rewriting the model-calling layer rather than configuring it. The second constraint is Postgres. DB_CONNECTION_STRING appears in the credentials block, and the example is a local Postgres, so a working database is part of the baseline. Teams that assumed a vector store would be provisioned inside the pipeline will find an external dependency instead. The third constraint is age. The most recent release listed is v0.0.9 from September 2023, and the cluster environment named in the README is Ray 2.6.2 with Python 3.9. Ray's Serve API and its LLM serving components have moved since then. The repository was pushed more recently than its last release, so the main branch may be ahead of the tags, but the absence of a newer tag means there is no versioned artifact to pin. The fourth constraint is that this is a notebook. Notebooks are good for narrative and bad for diffing, so if your team reviews changes through pull requests, expect friction.

Ray Serve versus a framework that hides the cluster

The closest alternative in spirit is a RAG framework such as LlamaIndex or LangChain, which package retrieval and generation into importable abstractions. The difference is where the complexity sits. A framework gives you a retriever interface and a query engine; you write less code, but scaling the indexing step across a cluster is left to you or handled by whatever integrations the framework exposes. This repository goes the other way. It keeps the stages explicit and uses Ray to distribute them, which is why the README can talk about scaling load, chunk, embed, and index as separate concerns. That is a real advantage when your corpus is large enough that embedding is a batch job rather than a loop, and a real disadvantage when your corpus fits in memory and you just want an answer. If your documents are a few hundred pages, a framework plus a single embedding call will be less work than standing up a Ray cluster and a Postgres instance. Choose this repository when the scaling problem is already yours.

Maintenance cost and the CC-BY-4.0 licence

The licence is CC-BY-4.0, which is a content licence rather than a software licence. That is consistent with what the repository is: a guide plus notebooks. Attribution is the condition, so reusing the material in internal documentation or a derived guide requires crediting the source. It is not a permissive software licence in the MIT or Apache sense, and it does not carry a patent grant. None of this is legal advice; if you plan to ship the notebook code inside a product, have someone check how CC-BY-4.0 applies to code as opposed to prose. On maintenance, the practical cost is tracking Ray versions yourself. The README pins a 2.6.2 cluster environment, and the tags stop at v0.0.9, so upgrades are your responsibility rather than something you get by bumping a version in requirements.txt. Budget for reading the notebook when you move Ray versions, particularly around the Serve application and the endpoint client. The evaluation harness is the part most worth preserving through those upgrades, because a scoring loop you trust is what makes any of the later changes safe.

Editorial conclusion

Adopt this repository if you already run Ray or Anyscale and want a worked example of chunk, embed, index, retrieve, and serve stages with an evaluation loop you can point at your own documents. Do not adopt it if you need a packaged library with a stable API, or if you cannot provide the Anyscale workspace, OpenAI and Anyscale API keys, and Postgres connection the setup expects. Before committing, check the notebook against the current Ray version you run, and confirm whether the DB_CONNECTION_STRING path still matches your Postgres deployment.

Official sources

1. Issues
2. License: CC-BY-4.0
3. ray-project/llm-applications on GitHub
4. README
5. Releases