Built an evaluated AI support agent for @SpotifyCares as the Hiver SDE Intern take-home: one structured Gemini call classifies a tweet into 12 intents, drafts a ≤280-character reply grounded in six BM25-retrieved threads from 27,627 real conversations, and decides whether to auto-handle or escalate — with deterministic money/security/legal/churn rules that can force an escalation the model cannot undo, and a release veto that replaces any draft carrying an uncited claim or unsupported link with a holding reply.
Treated the proof as the product: a 250-tweet golden set labelled twice against a written guide and adjudicated (κ 0.96 intent / 0.95 escalation), four baselines, paired-bootstrap 95% CIs, and a blind two-order LLM judge on a different model — the agent's replies beat the nearest historical reply by +1.42/5 (CI 1.15–1.67) across 200 messages, with the judge's 73.5% order consistency and its blind spots published alongside.
Froze a fresh 200-tweet holdout behind SHA-256 locks — sample, labels, threshold and every source file hashed before inference, and a runner that refuses to execute at any other code or label revision — then measured 0.938 escalation recall, 0.789 macro-F1 and 35% auto-handling; replaying the same 800 cached receipts through repaired guards lifted recall to 97.5% with zero new model calls, reported as a retrospective regression rather than fresh evidence.
Made every number reproducible without an API key: all 1,771 Gemini calls live in a committed SQLite replay cache keyed by model, prompts, schema and temperature, so one command recomputes the headline metrics in about 1.4 seconds on Linux and Windows CI — while the client survived the free tier with pooled-key rotation, per-key sliding-window rate limits, 429 cooldowns parsed from server retry hints and deadline-bounded retries.
Deployed on Vercel as a slim Python serverless function (FastAPI, BM25 index rebuilt from the committed corpus at cold start, two-slot concurrency, sanitized errors, no visitor prompts persisted) behind a React dashboard with a live playground, bring-your-own-Gemini-key support held only in sessionStorage, and HMAC-signed HttpOnly admin sessions gating the internal evaluation, golden-set and failure-mode pages.




