How to estimate AI features: a practical framework for product teams

Estimating a CRUD feature is a habit. Estimating an AI feature is an argument. The model works in the demo, fails 8% of the time in real use, and the 8% is exactly where your users are. Here is the framework we now use before we give a client a number on any AI feature.

The three axes we measure in AI feature estimation

Every AI feature we are asked to scope gets sized along three axes. Accuracy tolerance — how wrong can the output be before the user cares. Recoverability — if the model gets it wrong, what does recovery cost. Data exposure — what does the model need to see to do its job, and what is the blast radius if it leaks.

The matrix is not a scoring gimmick. Two features with identical happy paths can land a factor of three apart once recovery and exposure sit on the same page as accuracy.

The framework came out of estimates we got wrong, not out of a whiteboard session. The reel below is the studio where those post-mortems happen.

What we used to get wrong about AI estimation

Our first year of AI estimates were basically software estimates plus a fudge factor. We scoped the happy path, multiplied by 1.5 and called it a day. We consistently missed the eval harness, the fallback UI and the human-in-the-loop path. None of those are optional in production; all of them are invisible in a demo.

The one-page template we use for every AI feature

Every new AI feature has a one-page doc: task definition in one paragraph, accuracy floor as a single number, fallback UI in two sketches, human-in-the-loop path as a diagram, data footprint as a bullet list. If any of the five is hand-waved, the feature is not ready to estimate.