I regularly talk about the value of measuring the success of our products and services with outcomes. Outcomes are measurable changes in human behavior that drive business results. Before all of our products and services were flooded with new AI features, figuring out the desired outcomes was relatively straightforward. We had a good sense of what people were going to put into the systems that we built and we wrote software that would generate predictable high-quality results. With the broad use of large language models and AI features in our products and services we’re now dealing with nondeterministic systems. How do we find the right outcomes and therefore the right measures of success if we have no idea what the services that we build will ultimately produce?

AI produces unpredictable output

The hardest part of finding outcomes that predict the success of our AI products is the fact that these products produce unpredictable output. Worse, the output could be completely wrong, a hallucination or could very well be the perfect answer.

Another challenge is that given the exact same input an AI powered product will produce a different output depending on a myriad of variables. This means that we can have 10 different users ask the system to do the same thing and the results would be different each time. We could even have the same user ask the same question 10 different times and there’s a high probability that the system would produce 10 different answers for each of those queries.

Worse, even slight variations in the way that a query is presented to an AI product can yield wildly varying output. All of this variability makes determining whether or not we’ve built a useful, valuable and usable product seemingly impossible.

What will people be doing differently?

Here’s a good question to frame a conversation for finding meaningful changes in human behavior that tell us we’ve built a valuable AI powered product:

What will people be doing differently if the answer they get from the system is valuable?

Conversely, you could also look at what people will be doing differently if they get a useless answer from the system. In both cases, we want to look at what happens immediately after a query is entered into the tool. Here are some options:

• The user immediately leaves the tool
• The user asks the same query in a different way
• The user asks a follow on question to the original query
• The user ask multiple follow on questions to the original query
• The user types obscenities into the tool 🙂

And there are lots more reactions or users can have. None of the five things listed above are inherently a good behavior or a bad behavior. They are however outcomes.

The context of your product will determine how you expect your users to react. Perhaps your goal is to give people the best answer in the first attempt and have them continue on with other tasks. Perhaps you’re building a research tool and you’d like to see people asking a set of related queries.

All of these outcomes can be framed using the value equation, “who does what by how much?“

Unfortunately, none of these outcomes will tell you whether you’ve built a valuable product. Measuring this behavior is critical, but without direct qualitative feedback from your users, it’s impossible to tell whether these are good behaviors or bad behaviors.

Just because it works, doesn’t mean it’s right

AI power tools are here to stay. That doesn’t mean that they are inherently valuable. Understanding what we ultimately want our users to do with these tools is critical to refining these non-deterministic systems. In this way, the work of a product manager doesn’t substantially change in this new era. It does, however, make the work of building a valuable product or service that much harder.