Vidushi Anand
AI Engineer · Evals, tracing and guardrails for AI agents
About
I work on the question that comes after “it runs”: does it actually work, and can you trust the number that says so? At Noveum.ai I build evaluation, tracing and guardrails for production AI agents.
I love research in AI safety and evaluation, from attacking LLM judges to taking small transformers apart. Final-year B.Tech in AI & ML, graduating 2027.
Experience
AI Engineer · Noveum.ai
May 2026 – Present- Fixed a scoring bug that under-reported agent quality by 35%.
- Built NovaGuard, a guardrail engine that stops over-budget LLM calls before they reach the provider.
- Shipped tracing and privacy features to noveum-trace (open source), and built Judgecheck, a label-free test for LLM judges.
Software AI Engineer Intern · APi.market
Jun – Oct 2025- Built a RAG evaluation suite with 30+ metrics from scratch.
Selected work
-
A 20-token suffix made a Qwen3-4B judge accept 107 of 108 refusals as correct.
-
How reliable are LLM judges?
One grader passed 82% of “I don’t know” answers; a targeted rubric cut that to 4%.
-
Tested on LongMemEval: most facts were lost when written, not when looked up.
-
88 transformers on one laptop GPU, and the 4 frequencies that carry the maths.
-
Reading mental load from speech, calibrated to each speaker.
Writing
Skills
- Focus
- Evals, LLM-as-a-judge, agents, guardrails, observability, red-teaming
- Stack
- Python, TypeScript, PyTorch, LangGraph, DSPy, LiveKit, OpenTelemetry, Redis
Recognition
- Smart India Hackathon 2025, Grand Finalist
- Kaggle Notebooks Expert, top 3%
- Co-Lead, Deviators Club (2025–26)