Editor’s note: Rajiv Shah, PhD is a speaker for ODSC East this May 13-15. Be sure to check out his talk, “Hill Climbing: Best Practices for Evaluating LLM Applications,” there!

VCs, AI Engineers, and podcasts are all talking about Evaluation. I never thought I would see so much attention to evaluation. But folks are seeing that evaluations are central to getting applications working in production. But unlike bandwidth or GPU quotas, you can’t solve evaluation by throwing money at it. Excellence at evaluation isn’t about resources, it’s about a mindset. Let me share how I talk to AI Engineers about evaluation.

Gens you have is dragon

Generative AI allows for so many possibilities, it’s super exciting. However, that freedom to generate poetry, code, or financial analysis means it’s hard to tame. Yes, you can get good results very quickly. However, that last mile of improvement, the difference between “good” and “great”, requires discipline, work, and evaluations. There is no shortcut to taming the Generative AI dragon.

Evaluation as a Habit, Not a Handoff

Real evaluation begins before the first metric is logged. It’s the reflex to ask:

• What am I really trying to test?
• How might I be wrong?
• Which observations would actually change my decision?

This habit looks less like software engineering and more like scientific reasoning plus built‑in fallibility. Drop in a PhD sociologist (fluent in hypothesis testing, clueless about transformers) into your GenAI team, and they’ll often design sharper experiments than the Stanford CS grad who can hand‑roll CUDA kernels.

Why? Because the sociologist is happy living in the grey of uncertainty, poking assumptions, and treating every result as provisional.

Comfort with Grey

The evaluation mindset isn’t binary. It rarely moves in a straight, incremental path. Instead, it maps out the shades of grey — strengths, weaknesses, trade‑offs. That can feel alien to developers who thrive with the structure of computing systems, where correct and incorrect are black and white.

Comfort with grey doesn’t mean technical ignorance. It’s imperative to understand the design of technical systems for effective evaluation. Engineering at its heart is understanding that design choices have tradeoffs. If you are working on a RAG use case, you still need to know how embeddings feed a retriever, how a Large Language Model (LLM) consumes context, and the tradeoffs around latency. But the evaluation mindset asks: How will upstream errors impact the later stages? What are the ways this pipeline can break? How will users be impacted?

Choose Metrics, Choose Futures

“Evaluation isn’t grading the final test — it’s deciding what kind of student you’re raising.”

Metrics are incentives. Optimize only BLEU, and your model will mimic phrasing. Measure groundedness and user satisfaction, and it will learn to cite sources and speak human. No metric is perfect, each plays its own role to align with your larger goals.

A Three-Step Plan for GenAI Evaluation

These are three concepts you should use in evaluation.

1. Have a Map — Understand the System You’re Evaluating
2. Understand the Forest (Global Measures) vs. Trees (Unit Tests)
3. Pick and Choose Your Tools Appropriately

Tools Are Not Saviors

A Damascus steel knife can help a chef; they don’t create one. Likewise, LLM evaluation harnesses, LLM as a judge, or natural language unit tests are useful tools, but require a mindset to make them useful for your application.

Moreover, the latest agentic frameworks or evaluation tools are similarly not going to solve your evaluation problems (and can even create more technical debt). The reality is that masterful evaulation can be accomplished with REST APIs and Excel. My advice is to always wait to adopt or purchase any tools.

You Can Tame the Beast!

If this essay works, you’ll leave saying:, “Evaluation is not a milestone, it’s an intellectual habit that guides everything we build.” Cultivate that habit!

Want to Turn Mindset into Motion?

Join my workshop at ODSC East for hands‑on exercises, the latest techniques, and a new perspective on evaluation that sticks.

Thanks to Anatassia Kornilova, Derek Thomas, Bertie Vidgen, and Aravind Mohan for feedback.

About the Author: