“Oh man, we are SO much more productive than ever with AI!” Heard that one before? I know you have. Hell, I’ve said it myself. A lot. It really does feel like productivity has increased tenfold since AI went mainstream. Part of that is the appearance of quality in AI-produced output. It looks good. It looks polished. It looks done. And tweaking it to be even “better” is so simple it feels wrong not to do it. The question is, why are we polishing untested assumptions? Just because it looks like high-quality progress, AI-generated output doesn’t come with a guarantee of certainty nor validity. 

Teams over-invest in refinement because polished outputs look like “real work”

As your teams create artifacts for their work with Canva, Adobe tools, Figma, Lovable, Bolt or any other prompt-based tool (aren’t they all prompt-based now?) they’ll seem to be moving faster. Their cycle times between iterations will shorten. Why? Because their leaders expect it. Yet each iteration of an AI-produced deliverable is just a step further in a guessing game based on unvalidated assumptions. Refining the base output of an LLM-powered tool is a shot in the dark. The main difference is that it looks good. Why validate it if it looks “done”? 

This is particularly problematic in product design and development work. We can produce approximations of a user experience in a few minutes. An hour or so more of prompting and the refined experience looks ready to go into development. But wait! How do we know it’s a high-quality user experience? How do we know it will solve our business problem and meet the needs of our customers? We don’t – at least not with certainty. All we know for sure is that it looks good. 

Experiments prove quality with certainty

Rather than spending cycles refining an initial design, use that time to learn. Take the output from your favorite AI tool and show it to customers. Walk them through the experience and gather their feedback. Understand how they complete these tasks today and how your idea makes their activity better, faster, more efficient, delightful, etc. Take that data and then go back to prompting. The certainty that comes from doing great product discovery work can help you iterate your prototypes with greater confidence. That’s where progress is actually made. Once the mechanics of the user experience are refined your team can focus on the polish. And, to be clear, the visual design is also based on unvalidated assumptions and should also be tested. 

Here’s a pro-tip: Next time you’re generating a prototype or experience in your favorite AI tool, prompt the AI to create a grayscale output. Deliberately remove the polish from the experience. This way both your reactions as well as your customers’ and your stakeholders’ will focus more on the customer journey and less on the finished product.