Albertson’s, the US grocery chain, told the Wall Street Journal in August 2026 that shoppers who used its AI shopping assistant spend 10% more when using the conversational search bot and up to 26% more when they use the robust version of their assistant. Jill Pavlovich, SVP of digital shopping experiences at Albertsons was quoted in that article saying, “When they’re using a more comprehensive experience, they’re adding even more [items to their basket] because they’re not forgetting items.”  That’s my emphasis there at the end. Not forgetting the milk is an outcome. It is a human behavior but it’s not one that is easily measurable on its own on the grocery store’s side of the transaction. The store doesn’t know you forgot the milk. Only you do.  

Product managers are being pushed to redefine what a PRD is in the AI era. Consensus seems to be growing that evals are the new PRD. In fact two different publications came out recently with that exact headline. One was from Ameya Bhatawdekar at Braintrust back in March and the other was from Lisa Murkin over at Mind the Product. OpenAI’s CPO Kevin Weil seems to agree. He’s quoted in the Braintrust piece as saying, “Writing evals is the most important thing a PM can do in the AI era.” This makes sense. Trying to predict the right requirements in advance for a system that delivers different outputs per user, prompt and model in a PRD is nearly impossible. Evals, however, can do this and should. 

Here’s the interesting part though. In Albertson’s case above an eval would have told them whether the assistant is returning well-formed, accurate and on-brand answers to shopper queries. It’s unlikely that an eval would have told the Albertson’s team about shoppers forgetting items because this is a quality of the shopper not the system. And evals focus on system behavior. Somewhere along the way from traditional PRD to AI-era PRD, more than likely not on purpose, we’re losing the customer in the conversation (again). 

While I’m not going to get into how to build an eval suite (read Teresa Torres’ piece for that), I do want to cover what should be attached to that eval suite to ensure that we don’t lose the customer in the conversation and, in turn, build more useful AI customer experiences. Let’s dive in. 

Why an eval is a definition of done and not a product spec

Evals are tests with pass/fail criteria. This makes them an excellent part of the definition of done conversation. They are executable, run each time you commit new code, catch any regression issues and help settle quality arguments with data rather than opinion. Every one of those properties, though, focuses on the system’s behaviour. None of them is about the customer using that system. 

An eval suite scoring 94% tells you the feature behaves the way you specified 94% of the time. What it doesn’t tell you is whether anybody wants the feature or if the workflow to get to that part of the user experience improved or if the person paying for your service is spending more or less on average and why.

Both Murkin and Bhatawdekar, to their credit, warn about optimizing eval scores instead of user outcomes. Eval scores, ultimately, are an output. Or, at least, passing your evals is an output. It’s a feature of a system that is behaving more predictably and on brand. It’s still not measuring whether you’re delivering value or helping your customers make sure they have everything they need when shopping for taco night. 

What to attach to your eval suite before you call an AI feature done

So, how do we take the very useful eval suite you created and ensure it gives us a complete picture of “done” for our AI features? Here are 4 things to consider. 

1. The specific eval compliance score you will tolerate. This is more than just “the eval passes.” It’s a specific, written score that we agree is required to ship.
   
2. The outcome the feature is supposed to move. Add at least one sentence to each eval that names a specific customer behavior you expect your customers to be doing differently. Without this, the eval is measuring the quality of something that doesn’t necessarily add any value to the end user.
   
3. The person that will fix the eval/outcome failure. Name a specific individual who will take a look at why the bot isn’t hitting both its eval targets and its outcome goals and determine who should work on the fix. It may be a model issue, a ux issue or content problem. Put someone in the position to route those concerns properly.
   
4. The date the eval set gets refreshed from production. The eval dataset along with the desired outcomes reflect what you thought users would do. Live systems reflect what they actually do. Be clear when the data coming in from production will be used to update the existing eval and outcome set.

Three of the four of these should already be in your eval. It’s number 2, above, adding in the user outcome that makes this a product-ready definition of done. 

How to run this on a feature you are shipping this month

Let’s take the four items above and see how the Albertson’s example we started with changes. The eval the Albertson’s team comes up with can check that the assistant returns a real product in stock at the right store, formats the list correctly, and respects a stated dietary restriction. All of that can pass while the shopper can still easily forget the milk. The only mechanism to catch anything “forgotten” is basket size and that doesn’t tell us the item was actually forgotten. Maybe the shopper already had that item in their fridge?  

Getting the kind of basket size increase Albertson’s is seeing requires their shopping assistant bot to recognize what the customer is making for dinner this week and ask if they need to add ingredients currently not found in their basket. Basket size is the business result. Helping people not forget key ingredients is the winning user experience that keeps them coming back. Evals often miss that last part. 

Regardless of scenario, evals and outcomes are two important sides of the story teams need to deliver winning AI products.