The cv-santiago build script updates Reddit, GitHub, Discord and Twitter stats before it compiles

The build fetches social stats and pings a search engine before it compiles

The package is marked private and its version is 0.0.0, and the build script is a chain of eighteen steps in one line. In order: export the RAG chunks, ingest them, sync the prompt to Langfuse, embed the eval results, then four scripts that update stats, for Reddit, GitHub, Discord and Twitter, then generate the Open Graph image, validate the articles, run tsc, run the Vite build, generate a sitemap, generate an RSS feed, validate llms.txt, prerender, validate the prerender, and finally an IndexNow ping.

So a build is a publishing run. It reaches four external platforms, regenerates content derived from them, prerenders pages, and notifies an index. Every one of those steps needs a credential that a local checkout will not have, and a failure in any of them stops the chain before the type checker runs.

Two scripts exist as separate npm commands for the same work, with and without a fix flag: validate-articles and validate-articles:fix, and validate-llms-txt. That pairing suggests the validation is meant to be run in a loop during authoring, which is a reasonable design and one more reason the build is doing the wrong job.

The dependency style follows the same pattern. Nothing is vendored; every script runs through npx tsx with an explicit tsconfig, which keeps the toolchain loose but means a clean install and a build depend on the same npx path.

The architecture diagram stops at a half-drawn branch

The chatbot flow is drawn as a tree, and the tree is cut off. What the file shows is the request path: a user message reaches FloatingChat.tsx, then api/chat.js as a Vercel Edge function, which fans out into the system prompt from the Langfuse registry with a file fallback, a Claude Sonnet tool_use decision, and an agentic RAG branch used when needed, containing OpenAI embeddings through text-embedding-3-small, Supabase pgvector for semantic search alongside full-text BM25, and Claude Haiku for reranking and diversification.

Then the last line of the block is a branch marker and nothing after it. The RAG subtree is the deepest thing drawn, and the diagram ends in the middle of it, so the part of the system that decides what the model finally sees has no picture of itself.

The file table underneath carries the same information in prose form, which is more useful. Nine paths: the chat edge function, the shared RAG module doing hybrid search, reranking, cost tracking and intent classification, the prompt module with its registry and file fallback, the voice token endpoint that mints an OpenAI Realtime ephemeral token and rate limits, a separate RAG search for voice function calling, a voice trace endpoint with cost estimation, the React widget with streaming and quick prompts, a voice hook managing the WebSocket, audio capture and transcript persistence, and chatbot-prompt.txt as a fallback prompt, with production using Langfuse version 5.

The Tech Stack heading is empty and the stack is scattered across the file

There is a section titled Tech Stack in the project file, and nothing under it. The next heading is the chatbot architecture.

The stack itself is documented, just not in one place. The site is React 19 with TypeScript and a Vite build, from the description and from the vite.config.ts and tsconfig trio at the top level. The backend is a directory of Vercel Edge functions under api/, which is where chat, RAG, prompt, voice and the operations endpoints live. Storage and tracing are Supabase and Langfuse. The models are Claude Sonnet for the main decision, Claude Haiku for reranking and for the judge role, OpenAI text-embedding-3-small for embeddings, and the OpenAI Realtime API for voice. The architecture diagram animation is GSAP, and the search work is pgvector plus BM25.

Two smaller things about the repository itself. The language it is catalogued as is HTML, with an index.html at the root, which is what a Vite site looks like from outside. And the repository names no license and ships no license file, while the homepage is a public site, so what the code may be reused under is unstated.

The last commit is dated 2026-10-01 and there are no releases, so the only way to pin a version is by commit.

71 evals, and 16 of them are the only RAG ones

The test suite is the project's strongest claim, so the table is worth reading exactly. Seventy-one tests across ten categories, about 70% deterministic using contains, regex and word count checks, and about 30% judged by a model, Haiku.

By category: factual_accuracy 9, persona_adherence 4, boundary_testing 7, response_quality 7, safety_jailbreak 7, language_handling 5, rag_quality 16, multi_turn 5, source_badges 5, voice_quality 6. Those add to 71.

Two observations. RAG quality is the largest single category at 16 tests, which is where the complexity is, since retrieval can fail quietly in ways a string match never sees. And the categories marked Mixed rather than Deterministic are response_quality, rag_quality, multi_turn and voice_quality, which is 34 tests, or close to half the suite, being scored by a model instead of a rule.

So the CI gate is part deterministic and part a different model grading the first model. That is normal for this kind of product and it does mean a green gate carries two different kinds of confidence, which the category table is honest about and the headline number of 71 is not.

The same suite can be run outside CI with npm run evals, which points at evals/runner.ts, and the results land in evals/results/, which the dashboard embeds at build time.

A quality score below 0.7 becomes a test in the next commit

The closed loop is described in one line: trace, score online, and if quality drops below 0.7, generate a test automatically, which the CI gate then uses to block the deploy.

That is the most interesting mechanism in the repository, because it inverts the usual order. Failures in production produce the test cases rather than the tests predating the failure, and the threshold is a single number that decides what counts as worth remembering.

The scoring side has a command of its own. npm run evaluate-traces runs a batch evaluation with Haiku across three dimensions, quality, safety and intent, and the traces it consumes come from Langfuse rather than from a test fixture. So the loop has four moving parts: production traffic produces traces, traces are scored, low scores generate tests, and the tests gate the next build.

The retrieval half of the system has a diagnostic too, npm run diagnose:rag, whose stated job is detecting retrieval misses. That is the distinction the tooling draws, a trace can score badly because the model answered badly or because nothing relevant was ever retrieved, and only one of those is a prompt problem.

Six layers of defense, and one of them sends you an email

The prompt injection defense is six named layers: keyword detection, canary tokens, fingerprinting, anti-extraction, online safety scoring, and an adversarial red team. The last one has real-time jailbreak email alerts attached to it.

The red team is also a command. npm run adversarial runs scripts/adversarial-test.ts, described as 20 or more auto-generated attacks, which is the same shape as the eval loop: the threats are generated rather than written by hand, so the suite grows with the attack surface.

Three more places the same concern shows up. The operations dashboard has a Security tab listing the defense funnel, the safety distribution and the jailbreak list. The chats command can be filtered to jailbreak attempts alone with a flag, alongside a full-content flag for the last 50 conversations. And safety_jailbreak is its own eval category with 7 tests, listed as deterministic, so the blocking cases are string checks while the judgement cases sit in the model-scored categories.

Canary tokens are the layer worth understanding. A canary placed in the system prompt and watched for in output is a direct tripwire for extraction, and it only works if the prompt is the one being served, which is why the prompt lives in a registry with a file fallback rather than in a constant.

Voice mode runs on the OpenAI Realtime API and shares the retrieval path

The voice side is the same product with a different transport. It uses the OpenAI Realtime API, audio to audio, and it is stated at about $0.25 per session. Critically, it shares the RAG pipeline with text, so a voice conversation retrieves the same documents as a typed one.

The pieces are three endpoints and a hook. The voice token endpoint mints an OpenAI Realtime ephemeral token and rate limits it, which keeps the API key server-side and out of the browser. A separate RAG search endpoint exists for voice function calling, because the model has to retrieve through a tool call during a live session rather than through a request handler. A voice trace endpoint records the session with cost estimation. The client side is a hook that manages the WebSocket, captures audio and persists transcripts.

The operations dashboard has a Voice tab for sessions, the text and voice split, latency at P50 and P95, and cost per minute, and six of the evals are voice quality, all in the mixed category since audio has no regex to match.

Per-session cost is the number to keep in view. A portfolio that answers a question in text costs a fraction of a cent, and the same conversation in voice is quoted at about a quarter of a dollar, so the routing decision between the two is an economic one as much as a UX one.

Two SEO audit folders and a Playwright cache are committed to the repository

The top level is a working directory, not a clean package. Alongside the expected src/, api/, scripts/, evals/, tests/ and docs/ directories sit .seo-audit/, .seo-audit-v2/, and .playwright-mcp/.

Two versions of the same audit output, one with a v2 suffix, and a Playwright MCP cache directory. Both are artifacts of the tooling rather than source, and both are in version control, which means the audit history is reviewable, for better or worse, and the cache is downloadable by anyone who clones.

The rest of the layout is conventional and says something about the stack. Three tsconfig files, an eslint config, a Vite config, vercel.json for the deployment, a package-lock.json for npm, and .env.local.example for the secrets. There is no .env in the tree, which is the right default.

One file is worth knowing about: chatbot-prompt.txt at the root is the fallback system prompt, and the note next to it says production uses the Langfuse version. That arrangement means a deployment without registry access still has a working prompt, and a deployment with registry access silently serves whatever version the registry holds, which is the tradeoff for a prompt you can hotfix.

Editorial conclusion

cv-santiago is a portfolio first and a reference architecture second, and the second is where the reusable parts are: a production-shaped split between an edge chat function and shared RAG and prompt modules, a scoring loop that writes its own tests, and a voice path that reuses the same retrieval as text. Two cautions before you copy any of it. The build is a publishing pipeline, not a compile, so it needs credentials for four social platforms and pings a search engine on every run; anyone lifting the site should cut that chain down to the type check and the bundle. And the repository names no license and ships no license file, so the reuse terms of the code are unstated even though the site itself is public.

Frequently asked questions

Official sources

1. Issues
2. Project website
3. README
4. santifer/cv-santiago on GitHub