Someone asks a simple question. How many customers do we have. What did that job cost. Which sites are behind.
The answer comes back in seconds, formatted and confident. Then the person who asked spends the rest of the morning checking it by hand.
That is the honest experience of AI on business data right now, and it isn't a story about a bad tool. The tool worked. What failed was trust, and trust failed for reasons that have nothing to do with the AI model.
The agent has to guess
How many customers do you have?
1
Sales
One organisation, one relationship manager.
3
Finance
Three entities pay separately, each with its own account.
11
Operations
Eleven sites to service.
All three are right. Nothing written down says which version the business runs on.
Ask three parts of a business how many customers it has and you'll often get three numbers.
Sales sees one: a single organisation with one relationship manager.
Finance sees three, because three entities pay separately and each has its own account.
Operations sees eleven, because that's how many sites they service.
All three are right. Each is a reasonable answer to a slightly different question, and in most organisations there's nothing anywhere that says which version the business runs on.
An AI agent pointed at that estate has to pick one. It will pick, it won't tell you which, and the number will look exactly as authoritative as the correct one would have. That's the problem. Not a wrong answer, which you'd catch. A plausible one, which you won't.
Three things have to be true before any of this works, and none of them are AI work.
The shapes
What the business is made of, and how those things connect.
The rules
Where the calculations live, and the exclusions nobody mentions.
The shared vocabulary
One place those definitions live, in the words your people use.
Reporting layer
Questions you ask every month, in the same shape.
AI agent
Questions you didn't know you would have.
Two ways of consuming the same three layers. Siblings, not a hierarchy.
The shapes
What your business is made of, and how those things connect. Customers, sites, jobs, assets, people, and the relationships between them.
Most organisations have this implied across a dozen systems and written down in none of them. Each system was bought to do one thing well and made a sensible local choice about what a customer is. The choices never had to agree with each other, so they didn't.
Writing it down is less technical than it sounds. The hard parts are business questions nobody has been made to answer out loud. Is a site a customer. Is a variation a new job. When does a lead become an account. A couple of hours with the right four people in a room settles more of it than any tool will.
The rules
Where the calculations live. Today they're inside a report, or in column M of a spreadsheet, or in the head of the person who's run the month end for six years.
This is the heaviest part of the work and most of it sits on the client's side, because it can't be outsourced. Someone from the business has to say what each measure means, how it's worked out, which system is the one to believe, and who signs that off.
The part that catches people out is the filters nobody mentions. Almost every number a business quotes has silent exclusions behind it. Internal jobs are out. Cancelled work is out. The Australian entity is out. Anything under a thousand dollars is out. None of it is written down, everyone who builds a report knows it, and an agent doesn't. Left alone it will include what your people would have excluded and hand you a number that's technically correct and practically wrong.
“What was revenue this quarter?”
Every person who builds that report knows the answer leaves things out.
Silent exclusions
- Internal jobs
- Cancelled work
- The Australian entity
- Anything under a thousand dollars
None of it is written down. Everyone who builds a report knows it. An agent doesn't, so it hands you a number that is technically correct and practically wrong.
The shared vocabulary
One place where those definitions live, described in the words your people actually use, so a dashboard, a spreadsheet and an agent all resolve the same question to the same calculation.
This is the layer that turns the first two into something a machine can read. It's also the layer that makes an answer checkable, and checkable is the whole point. Without it every tool carries its own private version of what a number means, and the versions drift apart quietly until two of them turn up in the same meeting.
How you'd know it worked
Test one
Can you trace an answer back?
For any number it gives you, someone should be able to show the query that produced it and the definition that query used.
Fails when the path from answer to source isn't visible. You don't have a reporting tool, you have an opinion generator.
Test two
Has it been tested against answers you already know?
Write a set of questions where the business knows the correct answer. Run them. Record how many come back right, before go-live and periodically after.
Fails when the only evidence is the demo. A recorded accuracy figure against a fixed question set is the measure.
Where to start
Before this is an AI conversation at all
- Is your data modelled properly, with dimensions, facts and joins that work?
- Is there a named person for each area who can settle what a definition means and sign it off?
Both yes
You are closer than you think. The work in front of you is unglamorous, not large.
Either no
This isn't an AI project yet. It is a modelling project, or a definitions project, and that order costs considerably less.