A note about the scope of this article: this post is about the product feedback loop that PMs and their teams collect and act on. It’s not about model evaluation pipelines; if you’re looking for the engineering side of that, this evals explainer is a good place to start. The two work together, but they answer different questions.

In my house, this is a frequent scene. My wife will say something like, “Hey, I needed to figure something out asked Claude and everything it told me didn’t work. Can you try?” Then I fire up my Claude instance and ask it the same question albeit worded my way and get a completely different set of solutions to try. Maybe one of them works, maybe it doesn’t. And the pattern repeats. 

This is anecdotal and definitely not to scale. It’s also indicative of exactly how different the user experience and therefore the value of your AI product is to your various customers. Note that this isn’t even variable at the persona level. It’s variable at the individual user level and if you have a lot of users (well done, btw), this becomes a real challenge to ensure what you’re shipping is delivering value across the majority of that user base. 

When we built software whose behavior we specified, one person’s bug report described everyone’s product. With AI features, a piece of feedback describes one draw from a distribution  and most of our feedback techniques and mechanisms were built for the old way.

Why traditional product feedback breaks for AI features

Almost every feedback channel we rely on whether it’s online reviews, NPS, support tickets, the feedback widget etc, they all assume the thing being described is consistent. You could, in theory, reproduce the steps, see expected vs. actual behavior or take a screenshot that stands in for everyone’s screen.

AI features break that assumption in two directions at once. The same user gets different output on different days, and different users get different output on the same day. This is the same shift I wrote about in what “done” means when you’re shipping AI features. A feature whose behavior varies with every use is never finished in the old sense, and feedback collection becomes a much more complicated process that challenges how we determine whether we’ve shipped something “good” or not. The question changes. It’s no longer “what is the product doing?” It’s “what is the product doing across the distribution of what it does? and what are users doing about it?”

So how do we collect feedback on a product that’s different for everyone? What replaces “steps to reproduce”? Here are the three practices I’ve been recommending, and they’re all things your team can instrument this quarter.

Three ways to collect product feedback from an AI feature

1. Instrument the next action, not the opinion. The most honest feedback a user gives an AI feature is what they do immediately after seeing the output. Did they accept it as-is? Edit it heavily before using it? Re-query three times? Copy it out and then abandon the flow? These are correction and override events, and they’re observable and measurable in a way opinions aren’t. Pick one one of these behaviors to start, for example “the override rate” i.e., the share of AI outputs a user modifies or discards before using. I wrote about this from the OKR angle earlier this year. The same logic applies to feedback: what the user does next is the signal.

2. Review a broad distribution, not a single ticket. Once a week, pull a random sample of real outputs — 30 to 50 is plenty at the start — and have the team score them against a rubric so simple it fits on an index card (read: three questions, not thirty). It could be as simple as asking: Accurate? Useful for what the user was doing? Appropriate in tone? You’re not looking for the average score. You’re looking at the spread. The worst ten percent IS your feedback backlog. One caveat from doing this with teams. Review real outputs with real user context, not synthetic test prompts. The distribution you manufacture is likely to be kinder than the one your users are living in.

3. Capture the context, not the steps to reproduce. When feedback does arrive through the traditional channels, the reflex is to ask what the user clicked. With AI features the more useful question is what the model saw: the request, the surrounding context, roughly where in the quality distribution this output fell. Tag each piece of feedback with that context cluster and patterns emerge quickly usually. The complaints often aren’t evenly spread, they concentrate on specific kinds of tasks. That’s what turns “the AI is sometimes wrong” into something you can put on a roadmap.

All of this said, this does not replace talking to your customers. It changes what you ask about. Sit with five users and their last five real outputs and you’ll learn more than any survey will tell you.

What did the user do next?

The next time someone brings you feedback about your AI feature, before you act on it, ask what did the user do next? If nobody can clearly answer that question, you’re not collecting feedback on your AI products just yet. You’re collecting opinions about one draw from a distribution. Look at a broader distribution of actions, the context they happened in and what downstream behaviors that led to in a sample population of users. AI made the “vending machine” model obsolete and in doing so made user research harder. That’s an opportunity to not only build better products but to build a more customer-centric product organization.