An AI implementation that actually works rarely starts with picking a model. It starts with understanding the process it’s meant to support. Here’s the practical framework we follow: audit, pilot, production, ongoing support — and what actually happens at each stage.
Step 1: Process audit
Before the word “model” even comes up, you need answers to much more mundane questions: what documents or data flow into the process, where decisions get made within it, which of those decisions are routine versus which need expert judgment, and where the biggest time costs actually sit. A process audit means mapping all of this together with the team currently doing the work by hand — because they’re the ones who know where the process breaks and where the exceptions live that no instruction manual ever covers.
The output of an audit isn’t automation yet — it’s a clear picture of the process and an honest answer to which parts of it are actually a good fit for AI support, and which are better left alone.
Step 2: Pilot
A pilot covers a narrow, well-defined slice of the process — not the whole thing at once. It has one clearly defined success measure: for example, what share of a given document type gets read correctly and passed along without a human stepping in. Keeping the scope narrow lets you see quickly where the model performs well and where the edge cases show up — before you commit to a full rollout.
Just as important is what a pilot doesn’t do: it doesn’t yet replace a human in critical decisions. It’s a stage for gathering evidence, not a stage for handing over responsibility. A well-designed pilot also decides upfront what happens if the success measure isn’t met — whether the process goes back to the drawing board or a narrower scope gets tested instead.
Step 3: Production
This is where the gap between a demo and a system that runs day in, day out actually shows up — we covered that in more depth separately. Moving to production means adding quality monitoring, integrating with the systems the business already runs on, and drawing a clear line: AI reads, classifies, and proposes; anything involving money or a critical decision is handled by deterministic code.
Across the implementations we run (see our full AI offering), this split usually covers four areas: working with documents (reading and classification), assistants that operate on company data, decision automation with human escalation wherever a case is ambiguous, and integrating AI with the systems a business already has. The same rule applies in each: the model proposes, the system decides.
Working with documents means the model reads and classifies the content, while everything downstream — saving records, running calculations, sending notifications — is handled by ordinary code. An assistant on company data answers questions based on what the business actually holds in its own systems, not on the model’s general knowledge. Decision automation with escalation means routine cases get handled automatically, while ambiguous ones go to a person along with a reason the AI wasn’t confident. Integrating with existing systems means the implementation doesn’t require replacing tools the business already relies on — it plugs into them instead.
Step 4: Ongoing support
An implementation doesn’t end on launch day. Language models change between provider versions, input data drifts over time, and what worked well in month one may need adjusting by month six. Ongoing support means continuously tracking answer quality, responding to new edge cases as they appear, and managing cost deliberately as volume grows. In our experience, this stage — not the initial build — is where most of the effort in an AI implementation’s lifecycle actually goes, even though it rarely gets mentioned when a project is first being sold.
In practice that means things like a recurring review of a sample of outputs, watching whether the share of escalated cases creeps up over time, and updating validation rules once a genuinely new document or query variant starts showing up in the data.
An example: an accounting firm
Take a process that illustrates these four steps well, without referencing any specific client: an accounting firm serving many clients, receiving accounting documents every month in varying formats and quality. The audit shows where the most time goes into manually retyping data from documents. The pilot tests how well a model reads those specific document types. Production adds monitoring and integration with the firm’s accounting system, while the actual tax and accounting calculations stay with a deterministic engine, not the language model — exactly as in our own product Qkwit, where Claude reads the documents and code calculates the taxes and contributions. Ongoing support means continuing to track whether that reading stays accurate as the firm’s clients start sending new document formats.
Had the firm instead started by picking a model and rolling it out without an audit, it would risk building a system that handles the invoices used for testing well, but struggles with the ones that make up most of what actually arrives each month.
Why the order matters
Reversing this order — picking a model and rolling it out right away, before the process has even been mapped — is one of the most common reasons AI projects stall. Without an audit, it’s hard to know what’s actually worth automating. Without a pilot, it’s hard to gauge risk before it becomes expensive. Without a planned production stage, there’s no monitoring to catch a problem before it becomes serious. And without ongoing support, a system that worked well on launch day simply degrades over time. The order also reflects the cost of being wrong at each stage: fixing an assumption during the audit is cheap; fixing it after a production system is already running is far more expensive.
FAQ
How long does a process audit take before an AI implementation? It depends on how complex the process is and how many people need to be involved in the conversations — what matters most is including the team currently doing the work by hand, since they know the real exceptions.
Does a pilot immediately replace a person’s work? No. A pilot has a narrow scope and one success measure — it’s a stage for testing where the model performs well, not a stage for handing it responsibility for decisions.
What exactly does “ongoing support” for an AI implementation involve? Continuously tracking answer quality, responding to new edge cases in the data, and managing cost as volume grows — work that continues long after the system launches.
Can AI fully replace the person responsible for a process? Not in our approach — AI reads, classifies, and proposes, while critical decisions and financial calculations stay with deterministic code and, where needed, a person.
Where should a company start if it doesn’t know whether it’s ready for AI? With a process audit — before any decision about a model or a tool, it’s worth understanding exactly where the most time disappears today and where the exceptions live.
Let’s look at your process
If you’re wondering where to start with an AI implementation at your company, we’re happy to walk through the first step with you — the process audit. Get in touch.