A demonstration is easy to build. Point a capable model at some documents, put a chat interface in front of it, and within a week you have something that impresses people in a meeting. The gap between that and a system the business actually runs on is where nearly all the engineering lives.
Three problems that only appear after the demo
1. The data is in systems that were never designed to be queried this way
Business information sits in operational systems — a CRM, a platform database, a set of internal services — structured around the transactions they were built for. A question phrased in natural language does not decompose neatly into those structures. Bridging that gap is integration work, and it is the majority of the project.
2. A plausible answer is worse than no answer
In a consumer context, an approximate response is a minor annoyance. In a business context, a confidently wrong answer about availability, pricing or a client record has consequences that arrive later and cost more. Grounding is not a refinement to add in a second phase; it is the thing that makes the system usable at all.
3. An answer is not an outcome
The value of an interaction is rarely the text that comes back. It is what happens next — the record created, the lead routed, the workflow triggered, the follow-up scheduled. A system that answers well but changes nothing has moved work sideways rather than removing it.
Treat the AI layer as an untrusted client
The security model that scales here is the one already used for any other client of internal systems. The AI layer gets a scoped, mediated interface to business data rather than direct access. Its outputs are validated before they reach anything that writes. Its permissions are the permissions of the user on whose behalf it is acting, and no more.
The value of AI comes from integration with real workflows, data and decision-making — not from the model in isolation.
This has a pleasant side effect: it keeps the security review tractable. As the surface grows and more workflows are connected, the question is always the same one — what is this client allowed to read, and what is it allowed to trigger — rather than a new question per capability.
Evaluation has to be business-specific
General benchmarks say almost nothing about whether a response is useful to a particular client in a particular market. The evaluation that matters is built from the organisation's own cases: real enquiries, real constraints, and a judgement about whether the answer would have been acceptable coming from a person who knows the business.
That evaluation set is slow to build and unglamorous to maintain. It is also the only thing that tells you whether a change to the system made it better or simply made it different.
Related topics
- Enterprise AI
- Integration Architecture
- LLM Systems