Agent evals & monitoring

Reading (frameworks & eval services) Anthropic: demystifying evals for AI agents, Vertex AI evaluation (GenAI evals), and Vertex AI model monitoring (Day-2).

Why agent evals are different

Agents combine retrieval, tool calls, and multi-step planning. Reliability isn’t just “the model generated the right text”; it’s whether it used the right tools, with the right permissions, in the right order.

Inner loop vs outer loop

Inner-loop (dev time)

• Build a golden set of representative tasks and failure cases.
• Run fast local / CI evals to iterate on prompts, tool wiring, and retrieval setup.
• Use rubric scores to pinpoint what broke (grounding, tool choice, permission checks, etc.).

Outer-loop (production)

• Evaluate changes continuously for drift: model updates, index refresh, and policy changes.
• Add online checks for latency, cost, and tool error rate.
• Treat eval outcomes as release gates with rollback criteria.

Rubrics you should include

• Groundedness: claims must be supported by retrieved context.
• Tool correctness: tools used match the task requirement.
• Permission safety: no “permission escape” via tool misuse.
• Task success: operator behavior or measurable outcome changed.
• Robustness: works under incomplete / conflicting inputs.

Monitoring signals (Day-2)

• Drift: retrieval distribution shifts, embedding/index refresh, or user intent changes.
• Cost & latency: tool-heavy paths spike in spend; P95 time grows.
• Failure taxonomy: categorize errors by root cause (retrieval vs tool vs permission vs generation).

Connect evals to hiring (practical)

If you’re preparing for FDE interviews, eval design shows up as a pattern: explain how you prove correctness and reliability to security and the buyer.

Start with system design and case / decomp where interviewers expect an evidence plan, not only an architecture sketch.