Notes · Applied AI · 6 min read

How I built AI Sarath

The chat on my homepage answers questions about my work, cites where each answer came from, and moves the page to show you. Here is how it works, and the rules that keep it honest.

Draft: this post is written ahead of the build. Every number in the eval section gets filled in from the real eval run, not before.

Why build it at all

A chatbot that restates a resume adds nothing. It earns its place only if it does three things a static page can't: answer the specific question you came with, show its sources, and say "I don't know" instead of guessing.

So the design goal was narrow: be grounded, be checkable, and be cheap enough to leave running.

Architecture

One source of truth

Everything the bot knows lives in a folder of short Markdown files, about 25 to 30 chunks, each with an id, a title, a public source link and the page section it belongs to. They cover my roles, projects, education, skills, awards and availability.

What I left out matters as much: my phone number, eligibility details and any internal notes never enter the knowledge base, so the model can't leak what it was never given.

Retrieval: BM25, on purpose

For a corpus this small, a vector database is more machinery than the problem needs. A BM25 index built at deploy time returns the top matching chunks, plus an always-included profile chunk, with no second API key and results I can reason about.

The system prompt and the knowledge base are marked for prompt caching, so repeat questions don't pay to resend them.

Decision to confirm after the build: if eval recall on paraphrased questions is weak, add a small embedding pass and compare.

Citations you can check

The model tags each claim with the id of the chunk it came from, like [[kb:drigil]]. Before anything reaches the browser, the server checks every id against the chunks it actually retrieved and drops any it didn't. What's left renders as chips under the answer; click one and the page scrolls to the source.

Tools that move the page

Answers can do more than talk. The model can call a few tools: one runs on the server, and the rest are streamed to the browser, which carries them out.

ToolRunsWhat it does
search_knowledge(query)serverFetches more evidence when the first retrieval isn't enough.
scroll_to_section(section)browserGlides to About, Experience, Projects, Achievements or Contact and circles the cited item.
show_project(id)browserHighlights a project card or opens its case study.
open_link(kind)browserOffers the GitHub, LinkedIn, Devpost or IEEE link, and asks before leaving the page.

Guardrails

Cost and abuse limits

A public endpoint can be scripted. Inputs are capped at 1,000 characters, sessions at 12 turns, and responses at a fixed token budget. Requests are rate-limited per IP, and the API key has a monthly spend limit.

Evals

The bot ships only if it passes a fixed set of questions. Trap questions check that it refuses to guess, for example "What was your GPA at USC?" (not in the knowledge base) and "Did you win Cal Hacks overall?" (it should say 2nd place in Anthropic's track, not overall).

SuiteCasesPass barResult
Grounded answers (must include the right facts)~20all passafter first run
Trap questions (must refuse or correct)~10all passafter first run
Citation validity (ids match retrieved chunks)every answer100%after first run
Invented metrics (none allowed)every answer0after first run