Building an impressive AI demo takes an afternoon. Keeping a system running every day, on real data, taking responsibility for the outcome, is a different league entirely. Here’s what “AI in production” actually means, and why we see it so rarely in practice.
A demo has to impress. Production has to work
A demo has one job: work once, on a chosen example, in front of an audience that wants to see the “wow” moment. Nobody asks what happens when the document is unreadable, the model responds late, or the input looks nothing like the example prepared for the show.
A production system has to answer those questions every day, for every user, without exceptions. The difference isn’t in the model itself — the same language model sits behind both the demo and the production system. The difference is in what you build around it.
Monitoring and observability
In a demo, nobody checks whether the answer was correct — you can see it with your own eyes, because there’s only one example. In production, you need a mechanism that measures answer quality at scale: how often the model refuses, how often the result needs a human to step in, where the edge cases keep showing up. Without that, you don’t know the system is degrading until it fails visibly, which is too late.
Cost at scale
A single query to a language model is cheap when you’re showing it off once. Multiplied by thousands of documents, requests, or transactions a day, it becomes a real line item that has to be designed for deliberately — starting with which tasks genuinely need a model at all, down to how much context you actually feed it.
Model drift and errors
Language models sometimes get things wrong, and they sometimes change behavior between provider versions. A production system has to assume this from day one: a way to catch a bad answer before it reaches a customer or an accounting ledger, and a plan for what happens when the model isn’t confident.
Who’s accountable for the decision
This question never comes up in a demo, and it’s central in production. If the AI recommends something wrong, who’s accountable — the model provider, the company that built the system, or nobody? In a production system, the answer has to be clear from the start, not worked out after something breaks.
Our rule: AI proposes, code decides
Across the implementations we run, we hold to one rule: AI reads, classifies, and proposes — anything touching money or a critical decision gets calculated by deterministic code. A language model is excellent at making sense of unstructured input — a document, a message, a photo of a shopping basket. It isn’t the tool you hand the final tax calculation or a payment authorization to. In our view, this separation of roles is one of the main reasons most AI projects stall at the pilot stage: there’s no clear line between what the model proposes and what the system decides.
The proof: three products, one common thread
Rather than talk about AI in production in the abstract, we can point to our own products — all three run Claude in the production layer, not just in demos.
In Qkwit, our AI-powered accounting product for sole proprietors, Claude reads and extracts data from accounting documents, and the result feeds into a deterministic engine that calculates taxes and social security contributions. The model understands the document — the code calculates the liability.
In Taniej po Lek, our basket-based medication price comparison service running across nine countries, the Apteczkomat feature uses AI to recognize basket contents, hold a conversation with the user, and plan restocking. That’s a daily, production feature, not a conference demo.
In Brokik, our rental management platform available in 25 countries, AI supports adapting lease documents and pricing — in an environment where every market has its own rental law and its own contract templates, so the automation has to hold up across genuine variety, not just one idealized case.
What connects all three: the AI is working there every day, on real data, with monitoring and a plan for when it’s wrong — not just once, for a demo.
Why production AI is still rare
Many of the AI projects we come across stall at the working-prototype stage. The reason usually isn’t the model — it’s what’s missing around it: no monitoring, no clear ownership of decisions, no plan for cost at scale, no answer to “what do we do when the model is wrong.” These are engineering and organizational questions, not questions about whether the model is “smart enough.” That’s why we think AI that genuinely runs in production is still the minority case — not because models are weak, but because a full system rarely gets built around them.
What this means for your company
If you’re weighing an AI implementation, it’s worth asking from day one not “will this work once” but “what happens when this runs every day, on a thousand cases, with a different person on the other end.” That question leads to a very different set of design decisions — about monitoring, cost, the split between AI and code, and who owns the outcome.
FAQ
What’s the difference between an AI demo and a production system? A demo shows one chosen case. A production system has to handle every case, every day, with quality monitoring and a plan for errors — no exceptions and no pre-rehearsed scenario.
Can AI make financial decisions on its own? Not in our implementations. AI reads, classifies, and proposes; deterministic code handles anything involving money or a critical decision — that’s the rule we apply across all of our own products.
Why do so few AI projects make it to production? Usually what’s missing isn’t the model — it’s monitoring, a plan for cost at scale, and a clear owner for the outcome. These are typically engineering gaps, not model limitations.
How do you check whether a vendor actually has production AI experience? Ask about specific, running products rather than completed demo projects — and ask how monitoring and the split of responsibility between AI and the system actually works there.
Does AI in production cost more than a demo? Yes, because cost scales with volume — which is why a well-designed implementation decides upfront which tasks genuinely need a language model and which don’t.
Want to see how this applies to you?
If you’re wondering whether your process is a good fit for AI that will actually hold up in production, get in touch. We’re happy to talk through your specific case, no strings attached.