Earlier this summer, HBR published the results of a multi-year field study by Anne-Sophie Mayer, Elmira van den Broek and Tomislav Karačić that followed people who had to explain and defend AI-made decisions. These were broad decisions that included credit decisions at a German bank, hiring recommendations at a global consumer goods company and seed quality classifications at a biotech firm. The researchers found that these employees rarely passed the AI’s output along as it was, and instead they reinterpreted it, reformulated it and sometimes concealed it, depending on who they needed to convince. This makes a lot of sense to me. These folks had been made responsible for a decision they didn’t make, didn’t fully understand and had no or very little authority to change.

Organizations have moved AI into decisions that carry real consequences (e.g., pricing, credit, staffing, the commitments customer service makes on the company’s behalf etc) and yet the accountability structure underneath those decisions still assumes a human made the call. When one of those decisions goes wrong, the person who “owns” the system can say, accurately, that they didn’t make the error. Meanwhile the leaders who approved the rollout lose track of where and how these decisions are actually being made and find out on the day of an incident rather than in a review ahead of launch.

Why accountability for AI decisions breaks down

At Sense & Respond Learning we work with a lot of enterprise teams who are putting AI into their decision and approval workflows. One thing we are starting to see regularly is that organizations are indeed putting an approval step on the process diagram for the systems they’re building. Sometimes it even has a name next to it. But when we ask who owns the outcome of that decision, rarely can anyone in the room say exactly who it is. The approver signs off on what the model produced. The model’s development team points out that it performed within its accuracy targets (hi evals) and the business leader assumes the approver is the safeguard. Everyone seems to be doing their job and yes nobody is directly accountable for the result.

A second piece of HBR research, from Kropp, Bedard, Wiles, Hsu and Krayer, adds an uncomfortable wrinkle. In a randomized experiment they found that framing AI agents as if they were employees can shift accountability away from individuals, increase escalation and reduce the quality of reviews. I suspect that happens because the more human we make the system sound, the easier it becomes to treat it as the one who made the call. If you give it a name like “Earl” you can then say, “Well, Earl got it wrong.” Except Earl is an AI. You can’t hold an AI system accountable. Why? Because you can’t coach it, promote it or ask it in a post-mortem what it would do differently next time.

This is dangerous given the pressure to use AI, the hit-or-miss quality of AI-driven output and the unavoidable increase in corporate employees shipping AI slop to their colleagues without ever actually evaluating the output. 

So what do we do? How do leaders who have already approved AI into an operational decision path make sure there’s someone who can actually answer for it?

How to assign accountability before AI enters a decision path

Before any AI goes into a decision path, name the person who will deal with the consequences when it goes wrong – specifically. Once you have that person identified, make sure they have the two key things they’ll need to succeed in that role:

• The information they would need to identify a bad decision or sub-optimal output from the system
• The authority to do something about it, both in-flight and post-launch

Accountability is the combination of an identified individual and the supporting system (data + authority) that lets them respond when incidents come up. Without these two things you’ve just identified someone to take the blame when the AI does something “bad.”

If you can’t identify that person, or you find them and they lack the information or the authority, the decision point isn’t ready to be automated yet. This is true regardless of what the model accuracy numbers say. A highly accurate model, even if it’s already in the market, with no one accountable for its mistakes still dumps the risk onto whoever happens to be standing closest when it fails.

Design the AI review process around the accountable person

Most AI review processes we’re seeing right now are built to evaluate accuracy, drift and bias, which is necessary work for the data science team. However, it tells the accountable person very little about what’s actually happening to customers on the other end of the AI-driven decision. What if we designed the review process around the accountable human? The questions would be quite different. For example:

• Which decisions this month would I have made differently from the AI? 
• Where did I override the system, and was I right? 
• What would I need to see earlier to catch the next bad one? 
• What authority do I need to change how the system works when I find a problem?

That last question speaks directly to what Mayer and her colleagues observed. Employees reshape AI output to defend their credibility because they’ve been made responsible for a system they aren’t allowed to improve. Give them the authority to change it, a regular forum to raise what they’re seeing (the researchers recommend exactly this kind of ongoing dialogue) and recognition for questioning the output, and you reduce both the drive and the need to discretely massage the results. You also start to get a clear view of where decisions are actually being made. This, in theory, should be highly valuable to the leaders in your organization. 

A simple test for your next AI rollout

If you’ve approved AI into a decision path in the last year or so, pick one of those decisionsand ask these three questions:

• Who is accountable when it goes wrong? 
• Could they have seen it coming? 
• Could they have decided differently? 

If any of those answers is fuzzy or literally doesn’t have an answer, that’s a great place to start defining this new accountability process.

Give this a shot with one decision and let me know what you find.