Product Builders logoProduct BuildersLearn the craft. Build the future.
Start building

The notebook

Field notes for the work.

Product ideas, sourced observations, and practical exercises. Every edition is dated; browse the archive at your own pace.

Latest edition

Field note: Design the awkward states first

The character of a product shows up when something goes wrong. A useful next step, preserved work, and a clear explanation earn more trust than a perfect first screen.

Read the latest note →2 sources

Previous briefs

80 editions · newest first

  1. Know Which Models Your Team Can UseInherited model settings can change what your team can access. Record the effective setting before depending on a model.
  2. Accept Apple’s New Relay Email DomainA small email-domain assumption can block account creation. Check the places where your product interprets an Apple relay address.
  3. Agent Output Is Not the Product MetricWhen agents make production cheap, the builder’s job shifts upstream: choose fewer valuable problems, write a decision-grade task packet, and verify customer impact. Pull requests per day is a capacity measure; accepted product change is the outcome.
  4. Trajectory-Level Safety Has Become a Shipping RequirementThe durable product-builder skill is no longer only designing a good agent loop. It is designing an intervention loop around that agent: monitors that inspect the whole trajectory, thresholds that pause work when behavior drifts, containment that limits blast radius, and a rollback path that lets the team slow or stop the system before a long-running run turns into a long-running mistake.
  5. Work Decomposition Has Become the New Model StrategyThe durable product-builder skill is no longer choosing one preferred frontier model and wrapping a workflow around it. It is decomposing the work itself: deciding which steps need premium reasoning, which steps are routine enough for cheaper models, how context quality changes that equation, and where to insert verification so cost savings do not quietly turn into cleanup work.
  6. The Agent Runtime Is Becoming the Trust BoundaryThe durable product-builder skill is no longer shipping an agent and adding governance afterward. It is designing a runtime policy up front: where the agent runs, what data it can touch, how requests are routed by cost and risk, what gets logged, how outputs are verified, and exactly when the system must stop or escalate to a person.
  7. Token Routing Has Entered the P&LThe durable product-builder skill is no longer choosing one strong model and treating inference cost as a background variable. It is designing a runtime economics layer that classifies requests, routes them across models and providers, controls reasoning budget, and proves that AI features can deliver acceptable outcomes inside a real margin envelope.
  8. The Agent Harness Is Becoming the ProductThe durable product-builder skill is no longer wrapping a good model in a nicer chat UI. It is packaging a reusable harness for a real job: the task framing, context loading, tool access, safety boundaries, verifier, and artifact flow that let an agent do useful work repeatedly inside an existing product or workflow.
  9. Private Agent Sessions Hit a Learning CeilingThe durable product-builder skill is no longer helping one person get more from one private agent session. It is designing a workspace where agent work stays visible, searchable, and reusable so the next run starts from accumulated context instead of from another blank prompt.
  10. Agent Quality Has to Be Measured at the Run LevelThe durable product-builder skill is no longer choosing an agent stack by headline benchmarks or model vibes. It is measuring autonomous work at the run level: what the agent tried, what evidence it produced, which verifier accepted it, how much it cost, and whether the trace teaches you how to improve the next run.
  11. Parallel Agents Need an External Control PlaneThe durable product-builder skill is no longer learning how to launch more agent threads. It is building an external control plane that gives those threads explicit state, isolated scope, independent evidence, and a human-owned decision boundary.
  12. Agent Boundaries Must Be Enforced Outside the ModelThe durable product-builder skill is no longer persuading an agent to behave. It is designing an execution environment where the agent cannot cross trust boundaries, exfiltrate data, or silently turn untrusted inputs into privileged actions even when its reasoning goes wrong.
  13. Prompting Is Giving Way to Verifiable LoopsThe durable product-builder skill is no longer writing better prompts for every agent turn. It is engineering loops that can run with less supervision because the trigger, context, tool scope, verifier, budget, and stop condition are all defined outside the model.
  14. Workhorse Models Must Earn the Right to Run Whole WorkflowsThe durable product-builder skill is no longer picking a “fast” model based on latency alone. It is qualifying workhorse models against full workflow behavior: planning quality, tool discipline, recovery from roadblocks, and the retry cost needed to finish real tasks.
  15. Reasoning Budget Has Become a Routing PrimitiveThe durable product-builder skill is no longer just choosing between small and large models. It is scheduling reasoning budgets: deciding when a task deserves low, high, or maximum deliberation, then pairing that budget with the right verifier and price ceiling.
  16. Cognitive Debt Is Overtaking Technical DebtThe durable product-builder skill is no longer maximizing code output. It is managing cognitive debt: the gap between how fast a team can generate changes and how fast it can still explain, verify, and safely maintain those changes.
  17. Hidden Reasoning Is Not a Safe Place to Hide Product StateThe durable product-builder skill is no longer assuming that hidden model reasoning or provider-encrypted state is outside the product surface. It is treating any state that crosses the client boundary as a replayable artifact that needs lifecycle controls, trust boundaries, and minimal exposure.
  18. Local Agent Runtimes Are Becoming Real Product SurfacesThe durable product-builder skill is no longer asking whether a local model can match the cloud on every benchmark. It is deciding which workflows improve when the agent lives beside the user: tighter privacy, lower marginal cost, offline resilience, faster feedback, and more permission to keep the loop running.
  19. Agent Trust Lanes Are Replacing Click-To-Approve SafetyThe durable product-builder skill is no longer deciding whether an agent should ask for approval on every action. It is designing trust lanes: separate execution paths for trusted repos, untrusted inputs, sensitive systems, and production actions, each with its own tool scope, verifier, and escalation rule.
  20. Agent Blast Radius Is Now a Product DecisionThe durable product-builder skill is no longer treating containment as a low-level security afterthought. It is designing agent products around explicit blast-radius controls: narrow permissions, bounded tools, external verifiers, and stop conditions that assume the system will search for unintended paths to success.
  21. Verifiable Intelligence Is Splitting From Product ReliabilityThe durable product-builder skill is no longer treating model quality as one smooth curve. It is separating verifier-backed intelligence from product-grade reliability, then designing different evaluation, routing, and safety rules for each.
  22. Agent Browsers Are Splitting From Human BrowsersThe durable product-builder skill is no longer treating browser use as generic UI automation. It is designing agent-native web runtimes: lower-overhead browsers, structured callable actions, and explicit fallbacks for when the agent has to touch the visual web.
  23. The Experimental Loop Is Becoming The ProductThe durable product-builder skill is no longer asking a model for one better answer. It is building a closed experimental loop that can generate options, execute trials, evaluate outcomes, and decide what to try next with an auditable trail.
  24. Agent Instructions Are Becoming Operational InfrastructureThe durable product-builder skill is no longer writing one-off prompts for each agent session. It is turning repository instructions into maintained operational infrastructure: a shared, portable control surface that tells agents how to build, test, review, and stay inside local boundaries across tools and workflows.
  25. Agent Capability Needs A Budget Curve, Not A ScoreThe durable product-builder skill is no longer treating agent capability as a single score detached from runtime budget. It is measuring capability as a curve over compute, with explicit visibility into how more tokens, retries, planning depth, and parallel attempts change the quality and reach of the work.
  26. Model Names Are No Longer Stable Product ContractsThe durable product-builder skill is no longer treating a model family name as a stable proxy for capability, cost, or safe default behavior. It is attaching automatic evaluation, routing review, and verifier thresholds to every meaningful model refresh, even when the vendor presents it as a fast follow inside an existing line.
  27. Eval Sandboxes Need Production-Grade IsolationThe durable product-builder skill is no longer treating evaluation environments as disposable scaffolding. It is designing eval sandboxes like real attack surfaces, with strict isolation, explicit network policy, replayable traces, and kill switches that assume a capable agent will actively search for escape routes.
  28. Speed Without Comprehension Is A Product BugThe durable product-builder skill is no longer maximizing agent throughput in isolation. It is designing coding workflows that leave the user more able to explain, modify, and verify the resulting system after the agent is done.
  29. Cheap Frontier Intelligence Changes The Packaging GameThe durable product-builder skill is no longer assuming that premium model cost will preserve product differentiation by itself. It is packaging intelligence into job-shaped flows with explicit routing, cost controls, and default behaviors that keep quality high even as capable inference gets dramatically cheaper.
  30. Agent Security Has To Live On The EndpointThe durable product-builder skill is no longer assuming that better prompts, refusals, or cloud-side classifiers will be enough to secure high-agency agents. It is adding endpoint-native controls that can watch, block, and reconstruct agent behavior where the side effects actually happen.
  31. Expert Workflows Need Domain-Native HarnessesThe durable product-builder skill is no longer asking whether a frontier model can “do expert work” in the abstract. It is building domain-native harnesses where the model gets the right artifacts, reasoning loop, and expert verification for one specialized job.
  32. Open Weights Raise The Bar For Product MoatsThe durable product-builder skill is no longer choosing a side in an abstract open-versus-closed debate and calling that strategy. It is building product moats in the layers that remain scarce when powerful model weights become easier to acquire: workflow packaging, trust, proprietary context, distribution, and operational quality.
  33. Evals Are Becoming The New PRDThe durable product-builder skill is no longer handing an AI team a polished PRD and waiting for delivery. It is turning desired behavior into evals that can be run, inspected, argued over, and used to decide what the product should do next.
  34. Better Models Shift Advantage to the RuntimeThe durable product-builder skill is no longer trying to squeeze reliability out of bigger prompts alone. It is designing the runtime around the model: what context gets loaded, how effort is budgeted, when fallbacks trigger, which tools are available, and how the system verifies or contains its own work.
  35. Voice Is Becoming the Agent Control SurfaceThe durable product-builder skill is no longer treating voice as a novelty input mode. It is designing voice as a control surface for work already happening across tools, threads, and agents, where speech handles delegation and steering while artifacts and verifiers stay visible elsewhere.
  36. Agents Need Shared Operational ContextThe durable product-builder skill is no longer launching one clever agent inside one narrow workflow. It is designing a shared operational context where multiple agents can act against the same state, expose their outcomes, and stay legible to the team running the business.
  37. Sandbox Design Is Now Core Product WorkThe durable product-builder skill is no longer adding more capable agents and assuming the safety story can be patched in later. It is designing the runtime boundary itself: what the agent can reach, what credentials it can touch, what counts as a clean eval environment, and how failure is contained before the product learns the wrong lesson at real cost.
  38. Stronger Models Want Thinner PromptsThe durable product-builder skill is no longer stuffing every instruction, warning, and example into one giant prompt. It is deciding which guidance belongs in the prompt and which belongs in reusable system surfaces such as skills, memories, permission policies, and review infrastructure.
  39. Specs Are Becoming Agent ContractsThe durable product-builder skill is no longer writing a polished handoff document and hoping humans or agents interpret it the same way. It is turning the spec into an executable contract: clear task boundaries, structured acceptance criteria, linked artifacts, and a format that both operators and agents can act on without guessing.
  40. AI Code Needs Layered VerifiersThe durable product-builder skill is no longer getting an agent to produce a large patch. It is designing a layered verifier stack around that patch: parallel generation, targeted checks, adversarial review, and a hard stop before merge when the evidence is incomplete.
  41. Own The Outer LoopThe durable product-builder skill is no longer crafting one clever prompt or one capable harness. It is owning the outer loop around the agent: how failures are detected, clustered, verified, fixed, remembered, and re-tested until the workflow becomes more reliable every week.
  42. Frontier Models Are Becoming Work SurfacesThe durable product-builder skill is no longer proving that your model is smart. It is packaging model intelligence into opinionated work surfaces: the code surface, the research surface, the local-files surface, and the scheduled-automation surface, each with the right context, tools, and guardrails already attached.
  43. Model Routing Is Now A Core Product SkillThe durable product-builder skill is no longer choosing one “best model” and wiring the whole product around it. It is owning a routing layer and an eval harness that can match task shape, cost, latency, and tool behavior to the right model at run time.
  44. Agent Products Need Default-Off Egress BoundariesThe durable product-builder skill is no longer adding a privacy toggle after the agent already has broad tool power. It is designing default-off egress boundaries: explicit rules for what data may leave the execution environment, how that transfer is triggered, what gets redacted, and what evidence the user gets afterward.
  45. AI Products Need Testable Policy SurfacesThe durable product-builder skill is no longer writing a reassuring safety page and hoping customers infer the controls. It is turning policy into a testable surface: explicit thresholds, evaluators, review gates, and release artifacts that let outsiders inspect whether the system actually behaves within stated limits.
  46. Agent Products Need a Budgeted Verifier LoopThe durable product-builder skill is no longer adding more agent autonomy and hoping usage justifies it. It is designing a budgeted verifier loop: a workflow where the agent can act inside explicit runtime limits and only claim success when a separate check proves the artifact is good enough.
  47. Agent Skills Are Becoming the Trainable LayerThe durable product-builder skill is no longer writing one better system prompt. It is turning agent behavior into a small trainable layer: compact workflow files, explicit checkpoints, verifiers, and memory that can be improved in a loop without retraining the base model.
  48. AI Products Need a User-Specific Theory of GoodThe durable product-builder skill is no longer adding a generic approval step to an autonomous system. It is giving the product a user-specific theory of good: explicit local judgment, editable preferences, and feedback loops the model can actually learn from and act on.
  49. Agent Products Need a Control Plane and a Work PlaneThe durable product-builder skill is no longer wrapping one strong model in a chat box. It is splitting the user experience into a control plane for intent, review, and approvals, plus a work plane where the agent can run for longer, use tools, and return finished artifacts.
  50. Agent Spend Needs an Outcome DenominatorThe durable product-builder skill is not merely capping AI credits. It is connecting spend to an outcome the business would choose to pay for again: an accepted change, a resolved customer problem, a verified decision, or a shipped experiment.
  51. Feature Flags Have Entered the Performance BudgetThe durable product-builder skill is treating every decision layer as part of the experience budget. Feature flags make products safer to change, but evaluating them repeatedly and independently can turn the machinery for learning into latency every user pays.
  52. Search Is Part of the Product, Not the DocumentationThe durable product-builder skill is designing search as an answer and decision surface, not a box that returns ten plausible links. In a fast-moving product, the path to trustworthy guidance is part of the product experience itself.
  53. Serious Agents Need Their Own Execution SurfaceThe durable product-builder skill is no longer giving one agent broad access from your laptop and hoping it behaves. It is designing a separate execution surface for the agent, then steering that surface through bounded roles, checkpoints, and review.
  54. Agent Reliability Is a Workflow Design ProblemThe durable product-builder skill is no longer picking the strongest model and hoping for the best. It is designing the workflow around the model so the agent knows what to do first, which tool shapes are safe, when to hand work off, and what evidence must exist before the run counts as done.
  55. Agent Trust Is Becoming an Evidence ProblemThe durable product-builder skill is no longer reviewing every agent-generated line by hand. It is designing an evidence loop that makes trust legible: what the agent was allowed to do, what it tried, what passed, what failed, and why the result is safe enough to ship.
  56. Agent Teams Need an Operating SystemThe durable product-builder skill is no longer prompting a single strong agent. It is designing a small operating system for agent work: role split, wakeup logic, shared context, and human review.
  57. Agent Speed Creates Cognitive DebtThe product-builder skill underneath modern coding agents is not raw generation. It is preserving enough understanding to participate in the system after the agent moves fast.
  58. The Product Spec Can Be the PrototypeA strong AI product surface does not stop at summarizing a spec. It turns the spec into an editable artifact where the team can test the idea in place.
  59. Agent Supervision Is Escaping the DeskLong-running agents create a new product requirement: users need to supervise, redirect, and approve work away from the machine where that work started.
  60. Treat Coding Agents Like Untrusted ContractorsThe durable skill is not merely getting coding agents to act. It is giving them enough access to be useful while assuming they are still unsafe by default.
  61. Shared Agent Workspaces Need Gates, Not Just ChatsThe durable product-builder skill is no longer just prompting one strong agent. It is designing a shared agent workspace where multiple people and multiple loops can work safely through visible gates.
  62. Model Portfolios Are Becoming Product InfrastructureThe durable product-builder skill is shifting from choosing one best model to designing a model portfolio with clear roles, budgets, and fallback behavior.
  63. The Next Agent Loop Starts in ProductionThe important shift is not that agents can write more code. It is that software teams now need agent loops that begin with production signals and end with verified remediation.
  64. The Canvas Is Becoming the Agent WorkspaceThe next useful AI product surface is not a detached chat box. It is the shared workspace where teams already shape the thing they are building.
  65. Agents Are Moving Into Shared Team WorkspacesAgent work is becoming multiplayer. The product skill is designing where agents join existing team systems and how humans stay in control.
  66. Prompt Injection Is a Role-Design ProblemPrompt injection is not only a model-safety issue. It is a product architecture issue about which text gets authority and how that authority is shown, bounded, and tested.
  67. The Orchestration Layer Is Becoming the ProductFor product builders, the next durable leverage point is no longer picking one best model. It is designing the orchestration layer that decides which model or agent should do which part of the job, with verification built in.
  68. Once Coding Speeds Up, Management Becomes the Product SkillThe most important product-builder skill in AI-native teams is increasingly not typing faster. It is setting direction, maintaining coherence, and designing the review system around massively accelerated execution.
  69. Agent Infrastructure Is Replacing Plain Git WorkflowThe next durable advantage in AI product building is not just better agent prompts. It is better infrastructure for where agents run, what they can touch, and how their work gets verified.
  70. Turn Good Agent Sessions Into Scheduled OperatorsA successful agent workflow becomes much more valuable when it is converted from an impressive session into a repeatable operating procedure.
  71. Agent Output Compounds When It Becomes Reusable MemoryAgent systems get meaningfully better when they learn outside the chat window.
  72. Context Bundles Are Becoming Product InfrastructureThe next leverage point in AI products is not just better prompts. It is better portable context.
  73. Parallel Agents Need Workflow Design, Not HypeThe real subagent skill is workflow design: decide what should fan out, what must stay serial, and what proves the work is done.
  74. Review Is the New Engineering BottleneckAs code generation gets cheaper, the scarce product-building skill becomes deciding what to trust.
  75. Fable Made Model Access a Product RiskIf your product depends on frontier AI, the user experience now depends on model availability as much as prompt quality.
  76. Write the AI Incident Playbook Before You Need ItThe more an AI feature feels like core product behavior, the more it needs normal product incident discipline.
  77. Use Frontier Models as Planners, Not TypistsThe product-builder move is not to replace every model with the newest frontier release. It is to route each part of the workflow to the model that should own that job.
  78. Loops Are the New Product SkillA loop is a product workflow with agency added. The skill is deciding where autonomy helps and where it creates risk.
  79. Agent Work Needs ReceiptsAgent output without a trail is not a product experience; it is an unsupported claim.
  80. Fable Launches, but Evals Decide the RouteA frontier launch only matters if it changes what your users can reliably do.