Evals Are Becoming The New PRD

The strongest July 26 signal came from Digg Tech surfacing Lenny Rachitsky’s conversation with Anthropic product leader Dianne Penn. Her framing is unusually useful for builders: frontier products should feel frontier, PMs now need to sweat tokens as much as pixels, and evals are becoming more important than static requirements because they expose the jagged edge where capability and user value actually meet. The practical lesson is not that documents disappear. It is that product judgment moves closer to transcripts, traces, and scored outcomes. In AI products, what matters is not only what the feature is supposed to do, but what the system demonstrably does across messy real cases.

Traditional product artifacts assume software behavior is mostly determined by code paths and specifications. Frontier systems are different: behavior depends on prompts, context, models, tools, and runtime controls, which can all shift underneath the same user story. Teams that keep product management upstream of observed behavior will ship vague requirements and expensive surprises. Teams that anchor product decisions in evals will iterate faster because they can see where the system is good, where it is jagged, and what improvement would actually move user value.

Replace one fuzzy AI feature brief with an eval-backed product loop. Trigger: a feature request where the user value depends on agent quality, judgment, or long-horizon behavior rather than a deterministic rule. Context: gather a small set of representative real tasks, edge cases, and unacceptable behaviors before implementation. Tools: pair the feature spec with eval cases, trace review, prompt or harness variants, and a place to store scores over time. Verifier: define what “good enough” means in observable terms such as acceptance rate, error taxonomy, rubric score, escalation behavior, or time-to-useful-output. Budget: cap the number of scenarios, tokens, and review hours for the first loop so the eval system stays lightweight enough to use. Artifacts: produce the task set, scorecard, failure notes, and the change made in response. Stop condition: the feature only graduates when the evals describe the current behavior clearly enough for the team to make a shipping decision without hand-waving.