Most AI procurement follows the same script. Three vendors demo. One demo feels noticeably better. That vendor wins, the contract gets signed against a feature list, and eight months later the system handles half the volume everyone assumed it would, at an accuracy nobody ever wrote down.
The failure was in the buying, not the building. A demo is a performance optimized for exactly one thing: the demo. It runs on chosen data, along a rehearsed path, in front of people who won't be the daily users. Selecting on demos selects for the best presenter in the room.
What demos structurally can't show you
- The exception rate. Demos show the happy path. Your P&L lives in the other paths — the malformed document, the missing field, the customer who writes in two languages.
- Degradation over time. Every system looks good on day one. The question is month seven, after your formats drift and the vendor's attention moves to the next sale.
- Integration reality. "It connects to your ERP" in a demo means an API exists. It says nothing about your instance, your customizations, or your data quality.
- The cost of being wrong. A demo error gets laughed off. A production error lands on a customer, a regulator, or a balance sheet.
What written criteria change
An acceptance criterion is a sentence with a number in it, signed by both parties, tested on your data, with a consequence attached. The moment criteria enter a procurement, three things happen.
First, vendors self-select. The ones whose product can't hit the numbers find reasons to withdraw, which is the cheapest possible way to learn that.
Second, the conversation changes. Instead of arguing about features, you argue about thresholds and measurement windows — which is the argument that actually matters, held before signature instead of after failure.
Third, delivery incentives align. A vendor paid against criteria engineers toward your numbers. A vendor paid against a demo engineered toward the demo, and that work is already done.
Sample criteria for an intake-automation system
Here is the shape of an acceptance table we'd write for a document-intake system — the kind that classifies incoming files, extracts fields, and routes exceptions to people. Thresholds are illustrative; the real ones come from a measured baseline during scoping.
| Criterion | Metric | Threshold | Measured how | | --- | --- | --- | --- | | Classification accuracy | Correct document type | ≥ 97% | Weekly sample of 200, human-adjudicated | | Extraction accuracy | Critical fields correct | ≥ 99% on amounts and IDs | Reconciliation against source system | | Straight-through rate | Docs needing no human touch | ≥ 70% by day 60 | System logs, all volume | | Exception quality | Flagged docs that truly need review | ≥ 80% | Reviewer disposition codes | | Turnaround | Receipt to routed | ≤ 15 minutes, 95th percentile | Timestamps, all volume | | Auditability | Automated actions with full trace | 100% | Quarterly audit pull |
Two details matter more than the numbers. The measured how column removes the post-launch argument about whose measurement counts. And the straight-through criterion has a date on it, because ramp is normal — a system can be allowed to start at 50% as long as the contract says where it must be by day 60.
Where the criteria come from
You can't set a threshold without a baseline, and most organizations have never measured the manual process they're automating. That's why our engagements start with a scoping phase that times the current workflow before proposing anything — the acceptance criteria fall out of that measurement, and the monthly reporting under managed operations tests against them for as long as we run the system.
Ask every vendor in your next evaluation one question: what numbers will you sign? The ones who answer are worth your time.
If you'd like a criteria table drafted for a system you're scoping — from us or from anyone else — book a consult.