Last November I made the argument that AI was making Lean UX foundational. To be fair, it was a short post and didn’t go too deep into the details. Then, about a week ago, two new blog posts appeared that did a much better job than I originally did making this very same argument. 

In Building is cheap. Owning it is not., Ranjan Sakalley wrote that before the build starts product managers should “write down the prediction in one line,” because “a written prediction cannot be quietly revised.” That sure did sound a lot like writing a hypothesis prior to building your experiments to me. Not to be outdone, Pavel Samsonov, wrote on the same day in The fake double diamond is dead. The real double diamond is immortal., that teams should run experiments. He defined an experiment as “a deliberate task that a team undertakes to learn something it does not already know” and points out that an experiment with no real hypothesis and no planned end just leaves more clutter in the product. Again, this all sounded very familiar to me. 

Neither of these authors uses the words Lean UX yet both of them describe exactly that and, at least in the context of AI, they did a great job. Which I’m choosing to take as a compliment.

Complementing the growing consensus around the value of product discovery and lean ux in the AI era is the growing body of data. Atlassian’s State of Product 2027 report surveyed 1,000 product professionals, and 80% of them said AI is helping their teams ship faster yet customers aren’t seeing value any sooner. In the same survey 86% agree that deciding what to build matters more than ever. Writing down the “prediction” (read: hypothesis) of what you’d like to achieve before prompting anything into existence is where the bulk of the work is at the moment.

So how do you write that prediction? And what do you do with it once the feature you’re building runs on a non-deterministic model that gives every user a slightly different answer? I’m going to show you two things in this post, how to write the Lean UX hypothesis before you prompt the prototype into existence and how to turn that same hypothesis into the eval for that AI feature. LFG.

Why cheap AI prototypes make the Lean UX hypothesis foundational to PM work

Given the ease of creating a prototype, why bother writing the hypothesis first? The prototype looks good, impresses the boss and your clients and tells you very little about what you learned from it. Why? Because nobody wrote down why we built the prototype in the first place. It’s a tool for learning but without a clear sense of what we’re trying to learn, all we did was build a shiny object for stakeholders to click through. 

The hypothesis statement we wrote about in Lean UX was designed to give teams this missing context for their learning activities (read: prototypes). Box six of the Lean Product Canvas asks the team to complete one sentence before anyone designs or codes anything:

We believe that [this business outcome] will be achieved if [these users] attain [this benefit] with [this feature].

What Sakalley and Samsonov were referencing in their posts is the most important part of building prototypes and other types of experiments. How will we know if what we’re building is the right thing to build or is not adding any value to the customer? 

We can make the hypothesis statement even more specific and actionable, helping us determine whether we’ve chosen the proper solution for the customer by adding one more line:

We’ll know we’re wrong if [this observable behavior] doesn’t reach [this level] by [this date].

We do this first so that we don’t reverse engineer the story later when results start to come in from our experiments. We’ve come together as a team prior to building anything to agree on what success should look like.Then, once you have that agreement in place you can start to break the hypothesis down into its testable components.

How to turn a Lean UX hypothesis into an AI eval

One of the biggest challenges of building AI-powered products is that the very nature of the models that power these features vary their output from one user to the next regardless of the input. This is a big part of why finding outcomes for non-deterministic systems is harder than it looks. Teams building AI features deal with this through evals, tests that run the model against a set of inputs and score what comes back. I suspect most teams writing evals today have never connected them to a hypothesis. It’s simpler than you think because the hypothesis you wrote before the prototype already contains most of the eval:

• The users and the benefit tell you where the test inputs come from. Use real inputs from those users (last quarter’s actual tickets, queries or documents), because examples the team makes up, or a synthetic persona generates, mostly tell you how the model handles your own assumptions.
• The feature tells you what the cheapest prototype is, which is usually a prompt and a model with no interface at all.
• The “we’ll know we’re wrong” line becomes the passing bar, written as a rate because a non-deterministic system won’t give you the same answer twice (eight acceptable responses out of ten, say, with each input run several times).
• The business outcome stays outside the eval. An eval can tell you whether the output is good enough to put in front of customers, and only customers can tell you whether the hypothesis is true and valuable to them.

Here’s a hypothetical example, written generically so you can drop your own product into it. A support organization wants an AI assistant that drafts replies to billing tickets for its agents. The team starts off with a hypothesis statement:

We believe that we’ll cut time to resolution on billing tickets if our support agents can send an accurate first reply without rewriting it, with an assistant that drafts replies from the customer’s account history. We’ll know we’re wrong if agents send fewer than six in ten drafts with only light edits during a two-week pilot.

The eval comes straight out of those two sentences. Pull 200 real billing tickets from last quarter and ask two of your most experienced agents to write the scoring rubric (are the account facts right, is the policy right, would you send it). Run every ticket five times, and set the success threshold at eight in ten drafts get scored as “sendable” before anyone sees the tool. If the prototype you’re building can’t clear that bar, you’ve learned something valuable and have a good idea of what changes to make in your next iteration. And the cost of that learning? It was nearly nothing. At most it’s an afternoon and a few hundred API calls. If it does clear the bar, then we go back to the two key Lean UX questions. What’s the most important thing we need to learn next? What’s the least amount of work we need to do to learn it? 

You might then decide to run a two-week pilot test of the hypothesis itself with real agents on real tickets. This  moves us into the human behavior and outcome portion of the learning that an eval suite on its own can’t give you.

Write the hypothesis and the eval bar before your next AI prototype

I have good news about this way of working. None of it needs a new framework, an acronym or a maturity model. In fact, most of it has been sitting in the Lean Product Canvas for years waiting for the cost of building to drop low enough, it seems, that teams would finally feel comfortable using its full potential on risky work. 

Take the next AI feature on your team’s roadmap and, before anyone prompts anything into existence, write a Lean UX hypothesis for that feature. Make sure to add the line that tells you when you’re wrong, and turn that line into the passing bar for your eval. If the team can’t agree on what that line should say, you’ve lined up your next most important conversation and not a minute too soon. This might be the last cheap moment to have it.

Give this a shot on your next AI feature and let me know how it goes.