AI: Magic 8 Ball or Expert Intelligence?

Making AI useful needs Expert Intelligence and Constraint Layer™

 

By David Bernard, Managing Director, DBMT

​

Your AI tool or API sounds confident. It's always available. And the answers, while impressive, rarely survive contact with the real world.

​

There's a moment many teams hit after deploying an AI tool or API. The demos went well. The use cases looked promising. And now, a few months in, something feels off. The tool works — technically. But it hasn't changed how decisions get made.

​

The outputs are well-written. The responses are fast. But when someone tries to act on them, there's a gap. They need to verify. Reframe. Interpret. Fill in what the AI didn't know to include. The work hasn't disappeared — it's moved downstream, closer to the deadline, farther from where it would have been useful.

This is the Magic 8 Ball problem. Not that the answers are wrong, exactly. It's that they're designed to satisfy the question — not to survive the decision.

​

The power user trap

At this point, a reasonable objection surfaces: skilled users get much better results. And that's true. Teams with strong AI literacy — people who know how to frame problems precisely, layer context progressively, and challenge weak outputs — do produce something genuinely useful from general-purpose models.

But look closely at what that actually requires. Getting reliable, decision-grade output from a general LLM isn't a single prompt. It's a cycle. Frame the problem. Review the response. Reframe. Add constraints. Challenge the assumptions. Ask follow-up questions. Redirect when the model drifts. For a sophisticated analysis, this can take a dozen or more exchanges — and that's before any validation happens.

​

This is the dynamic that rarely surfaces in AI ROI conversations: the shift isn't from arduous work to easy work. It's from arduous drafting to arduous prompting. The cognitive burden moves; it doesn't disappear. And because prompting skill varies dramatically across any team, so do the results — creating a two-tier system where a handful of power users carry the quality of AI output for everyone else.

When productivity gains depend on a small group of people running Socratic dialogues with a language model, you haven't built a scalable system. You've built a specialist dependency — and one that produces inconsistent outputs depending on who's behind the keyboard on any given day.

​

Confidence is the tell

What makes this pattern so persistent is that AI outputs are genuinely impressive on the surface. Articulate. Structured. Delivered with an air of authority. In a meeting, a well-formatted AI-generated recommendation can hold the room — right up until someone with domain knowledge starts asking follow-up questions.

The real problem isn't that AI lacks intelligence. It's that AI, in most implementations, lacks constraint. It doesn't know the rules of your industry. It hasn't internalized the trade-offs your organization makes every day. It doesn't know what "good" looks like in your specific context — only what plausible looks like in general.

 

And plausible, at scale, gets expensive.

"A good answer is not the same as a usable one. Usable means it holds up under scrutiny from someone who has to live with the outcome."

​

What the gap actually requires

Most of the current conversation about AI focuses on models, interfaces, and prompts. Those matter — but they're not where value is created or lost. The gap lives between the AI generating an output and someone being able to act on it. And closing that gap requires something that most deployments haven't yet built.

It requires what we call an Expert Intelligence and Constraint Layer™.

 

This isn't prompt engineering, and it isn't a better interface. It's the system that sits between the AI model and the user — encoding decades of domain expertise, decision logic, and operational constraints directly into how the tool works. It's the difference between giving a brilliant generalist access to your problem and bringing in a seasoned specialist who already knows the rules of the environment.

will.

​

Building this layer isn't a configuration task. It requires a genuine understanding of how decisions are made in practice — not just in theory. The domain logic, the exception handling, the constraint hierarchies, the judgment calls that experienced practitioners make instinctively: all of it has to be surfaced, codified, and embedded in a way that holds up across the full range of situations the tool will encounter.

​

That's what makes it hard to build — and what makes it valuable once it exists.

​

"Why can't we just use ChatGPT and do this ourselves?"

It's the right question, and it deserves a direct answer.

You can. A skilled team, with the right AI literacy and enough time, can replicate many of the outputs that an expert-layer tool produces — through careful, iterative, Socratic prompting sessions. The quality ceiling is high. The problem is everything surrounding it.

​

First, the time cost. Reaching decision-grade output through iterative prompting isn't a five-minute exercise. It's a discipline. For complex, high-stakes guidance, you're looking at extended sessions — framing, challenging, redirecting, and validating — before anything is usable. Multiply that across a team, across a quarter, and the efficiency case for "free AI" starts to erode quickly.

Second, the consistency problem. The output you get from a general LLM is only as good as the session that produced it. Change the user, change the day, change the framing slightly — and you get a materially different result. In environments where consistency, auditability, and defensible reasoning matter, that variability isn't a minor inconvenience. It's a structural risk.

​

Third — and this is the point most often missed — the expertise problem. The Socratic prompting cycle only works if the person running it already knows what good looks like. To challenge the model effectively, to recognize when it's drifting, to know which constraints matter and which outputs are subtly wrong: that requires deep domain knowledge. Which means your best AI outputs still depend on your most experienced people doing the prompting. You haven't democratized access to expertise. You've just made experts faster.

"The Expert Intelligence and Constraint Layer™ doesn't replace expert judgment. It makes expert judgment available to everyone — without requiring everyone to be an expert."

​

When the expert layer is built well, the Socratic cycle isn't a workaround anymore — it's unnecessary. The system already knows how to frame the problem, which constraints apply, and what a decision-ready output looks like. That knowledge doesn't live in the user's head. It lives in the tool. Which means a junior analyst and a senior director are working from the same foundation — and producing consistent, reliable, defensible guidance at the first pass.

That's not a productivity improvement. That's a structural change in how expertise flows through an organization.

​

A useful test

If you want to quickly evaluate where your AI actually sits on this spectrum, there's a simple question: can someone on your team use the output immediately, without interpretation or rework? Not occasionally — consistently, across users and use cases.

​

If the answer is yes, you've built something real. If the answer is "sometimes, when the prompting is right," you're managing the gap rather than closing it. And if the answer is "our best people get good results" — that's the power user trap. You've made AI output quality dependent on the scarcest resource in the organization.

 

AI doesn't fall short because it lacks intelligence. It falls short because intelligence, without an embedded Expert Intelligence and Constraint Layer™, produces answers — not guidance. The organizations getting the most from AI right now aren't the ones with the best models or the most sophisticated prompt engineers. They're the ones who've done the hard, domain-specific work of encoding expertise into the system itself — so that expertise is no longer a bottleneck. It's infrastructure. That layer is more buildable than most people think. And it's worth the conversation.

​

If this is a challenge you're navigating — or a gap you've already identified— I'm always interested in comparing notes with practitioners who are thinking seriously about the distance between AI potential and AI value.