The AI engineering skills your team needs, from the production floor

1. Building AI systems you can measure

AI outputs are not predictable the way ordinary code is, so the core skill is turning a model's behaviour into something you can measure and steer. In practice that means evals: a test set of real cases with known right answers, run against every change.

When Native built the system that decides endorsements for TIE, the test set was the MGA's own historical decisions. An agent passes when it reaches the decision an underwriter reached, or refers the file. Every change to a prompt, a model or a rule runs against that set before release, and samples of live decisions are reviewed every week. Teams that skip this step ship demos; teams that do it ship systems.

2. Software fundamentals still decide the outcome

Most of what makes an AI system work in production is not the model. It is reading messy documents reliably, integrating with the system of record, handling the case where an upstream service is down, keeping secrets out of prompts, and logging every action so it can be audited.

Engineers who understand those trade-offs give agents better instructions and catch the poor choices a coding agent makes on their behalf. The fundamentals are now the leverage, not the overhead.

3. Working with coding agents

Coding agents are now how good teams write most of their code. The skill is knowing how much to direct them: give them the context they need, a clear spec when it matters, and a way to check their own work, such as tests they must pass. Then review what comes back.

Native builds this way on every engagement, with guardrails that matter in a client's systems: agents open draft pull requests and never merge their own work, review runs on every change, and agents never touch production data directly.

4. Shaping the build

As agents get better at building to a spec, the scarce skill moves upstream: deciding what the spec should be. That takes product sense and business context, which is why Native's engineers work inside the client's business. They sit with the underwriters, the order-entry team or the finance team, map how the work really moves, and design the system around that, not around a requirements document.

It also means knowing when to ship a quick version to learn from and when to slow down because a decision carries real risk.

How to build these skills in your team

• Start with one production system, not a training course. Skills come from shipping one real workflow with evals, review and a written authority line.
• Make evals a release gate. No change to a prompt, model or rule ships without running the test set.
• Put coding agents on the backlog first. Small fixes as draft pull requests teach the team how to direct and review agents with low risk.
• Send engineers to the work. A day with the team whose process you are rebuilding is worth a week of requirements meetings.
• Write down what you learn. Practices for working with agents go into the team's playbook so new hires start with them.