Why most AI pilots never reach production, and what the ones that ship do differently

The demo proves the model, and production asks about everything else

A pilot shows that a model can read an email, summarize a file or draft a reply. That was rarely in doubt. What a pilot doesn't show is whether the output lands in the system the business runs on, whether anyone trusts it enough to stop checking every line, and who answers for it when it's wrong.

Before Native's work with TIE, the MGA had already tried automating endorsement review in ChatGPT, hit the limits and seen the potential. That's the usual starting point. The team knew the model could do the reading. What was missing was everything around it: document handling for messy packs, a rules engine, a refer path to underwriters and a way to know the system was right.

1. Nobody owns it once the pilot ends

Pilots are often run by an innovation team or a single enthusiast, and the people who own the workflow are spectators. When the pilot ends, the sponsor moves on and the operations lead was never asked to take it.

At Avalara, AI adoption had depended on individual champions, and initiatives were fragmented across teams. The fix was structural: a forward-deployed engineer aligned to each department and accountable for adoption and delivery there, with one intake model for every idea. Ownership sat with a named person inside the department whose process was changing, which is why the work outlived its first sponsor.

2. It never writes to the system of record

A pilot that produces a draft in a chat window still leaves a person to retype the result into the ERP, the policy system or the database. The hours stay where they were, and the pilot gets judged on a saving it never delivered.

At USCAPE, every order had to land in a FileMaker database with 121 tables, 714 scripts and a 255,000-row price list, in its exact naming. The agent was only worth building because approved orders push into FileMaker with retries that can't double-post and a full audit trail. That integration was most of the engineering, and it's the part a pilot skips.

3. There is no way to tell whether it is right

Without an eval set, quality is judged by whoever looked at the last few outputs. One bad answer in front of the wrong executive ends the project, and one good week gets it promoted to production without anyone knowing its error rate.

The builds that ship treat a test set of real, historical cases as a release gate. On the USCAPE order agent, a 12-point checklist built from the operations lead's requirements checks every draft, and any field below 70% confidence is flagged before it reaches review. The AI adoption playbook sets out how to build that gate.

4. Nobody wrote down what the agent may decide

Risk and compliance teams stop pilots when they can't see the boundary. If it isn't written down which actions the agent takes alone, which it prepares for a person and which it never touches, the safe answer is to keep a human on every item, and then the pilot has saved nothing.

TIE's system decides endorsements where the rules settle the case and refers everything else to an underwriter with the analysis already written. Because that line was explicit, 74% of endorsement decisions now need no underwriter, and the ones that do are worked within the hour. Written authority is what lets a team stop double-checking.

5. The people whose work changes were never brought in

An agent changes someone's job. If the person doing the work today isn't the one who reviews the agent's output and shapes its rules, they have every reason to find its mistakes and none to fix them.

On the order desk, the person who used to key orders became the reviewer, approving drafts beside the source email. At Avalara, 40 GTM leaders were trained in person and optional platform training reached a 65% completion rate. Adoption was planned and tracked as work, the same as the build.

How to get a pilot into production

• Name the owner before the build. One person in the department whose process changes, accountable for the result after go-live.
• Scope the write first. Decide which record in which system the agent's work ends in, and build that integration as part of the pilot.
• Build the eval set from history. Real past cases with known right answers, run before release and after every change to a prompt, model or rule.
• Write the authority line. List what the agent decides, what it prepares for a person, and what it never does, and enforce it outside the model.
• Make today's operator the reviewer. Their checklist becomes the system's rules, and their corrections become the next round of evals.
• Agree the measure up front. Hours saved per year, cycle time or decisions without a person, reported to the owner every month.