How to measure applied AI

In May, Harvey released a benchmark of more than 1,200 legal tasks written by practicing lawyers across 24 practice areas. A task passed only if the work met every criterion in its rubric. The best model completed 7.1 percent of them. The same models sit near the top of the public tests most people use to compare them.

That gap is the subject of this playbook. In why evals matter we made the case that public benchmarks cannot answer an enterprise's question, and that the measurement you build from your own work is an asset. This piece is the practical companion: how to measure an AI system that is doing real work inside a company, what to measure, and how to keep measuring once it is live.

Why applied AI needs its own measurement

Four facts from the past year explain why measurement inside an enterprise is a different discipline from measurement in a lab.

Real work is much harder than the leaderboard suggests. On a public index of real freelance projects, the best models deliver about 21 percent at a quality the client would accept. On a set of realistic customer relationship management tasks, agent success falls to about 35 percent once the conversation runs over several turns. On long, multi-step work, a model that looks nearly perfect on short questions is often wrong somewhere along the way, and a single wrong step can spoil the result.

The best model changes from task to task. Rogo, which builds AI for finance, tested ten models on 928 questions written by former bankers and investors. The top three finished within 0.3 points of each other overall, yet each led a different area: one on capital structure and mergers, one on earnings quality, one on private capital and forecasting. Simply routing each workflow to the model that was best at it added 4.5 points over the best single model. Cost per task ranged from about two cents to about 1.26 dollars. Ramp reports the same pattern across invoices, financial statements and policy checks: a different model is the cheapest one that is good enough for each.

A better model can make your workflow worse. An independent test of 711 enterprise workflow tasks, graded on whether the business records ended up correct, found that every configuration of a newer frontier model scored 4 to 9 points below its predecessor, even though the newer model led on most public benchmarks. The newer model found the right information and then kept going, creating duplicate records and writing values nobody asked for. Separately, researchers found that upgrades which improved the average score by up to 7 points still made up to 8 percent of individual cases reliably worse. An average can rise while the cases you care about fall.

People's sense of whether it is working is unreliable. In a randomized study, developers overestimated how much AI sped them up by about 40 percentage points. In a study of nearly 6,000 executives published in February, more than 80 percent reported no measurable effect of AI on productivity yet, despite widespread use. Self-reported savings are a signal of enthusiasm, not a measurement.

There is also a practical pressure. Model providers now retire models on a schedule measured in months: one major provider announced sixteen deprecations in 2026 alone, and another gives about two months' notice. Every retirement forces a change, and every change needs to be tested. Even evaluation tools move: one provider's own evals product shuts down at the end of November. The measurement has to live with you, not with any vendor.

The playbook

1. Measure a workflow, not a model

The unit of measurement is a specific piece of work: extract the fields from this claim, draft this credit memo, resolve this support request, flag this expense for audit. Write down its inputs, the steps it takes, the systems it touches and what a finished result looks like. Everything that follows is measured against that definition.

This sounds obvious, and it is the step most often skipped. "Is this model good?" has no answer. "Does this workflow produce claims summaries that our adjusters accept without changes?" does.

2. Start from real cases and read them

Before writing a single automated check, collect real examples of the work and read them. The method that has become standard among practitioners is simple: take around a hundred real cases, run them through the system, read every output, and write a short note on the first thing that went wrong in each. Then group the notes into categories and count them.

This does two things. It shows you what actually fails, which is rarely what the team expected. And it gives you a ranked list of what to measure, so the effort goes to the failures that happen most or cost most.

Add hand-written cases for the rare failures that matter: the request the system should refuse, the document with a missing page, the customer asking for something the policy does not allow, the task that cannot be done at all, where the right answer is to stop and hand it to a person. Production traffic under-represents exactly these, and they are often the most expensive to get wrong.

Keep adding cases. Teams that do this well review a fresh sample of real work every few weeks and add every new failure they find.

3. Define success as the business outcome

Grade what the business cares about, not how the answer reads. A support conversation succeeded if the customer's problem was solved, not if the reply was polite. An agent that updates records succeeded if the records are correct afterwards, and failed if it wrote the right answer in its message but the wrong value in the system. The best benchmarks for agents work this way: they check the final state of the database, not the agent's account of what it did.

Be strict about definitions, because headline numbers move with them. Intercom changed which conversations count in its resolution rate this summer, and the rate went up while the share of work the AI actually handled stayed the same. A customer who leaves a chat without replying may have been helped, or may have given up. Decide in advance what counts, write it down and keep it fixed.

4. Choose graders by type of output

Different outputs need different ways of checking them. Use the simplest check that works.

Output	How to grade it
Structured fields, amounts, dates, codes	Exact match, or a numeric tolerance
Required elements, forbidden phrases, format	Rules and pattern checks
Changes to records or systems	Check the final state of the system
Citations and sources	Check each claim against the cited source
Open-ended writing, judgment, tone	An AI grader checked against your experts, or expert review

Prefer pass or fail over scores out of five. A clear yes or no forces a decision about what good means, is easier for experts to agree on and is easier to track. Where a task has several parts, check each part separately and report how many passed.

An AI grader is a measuring instrument, and it needs calibrating before you trust it. Have your experts label a few hundred examples, build one grader per failure type, and measure two things separately: how often it catches real failures, and how often it correctly passes good work. Shopify's first grader for its merchant assistant barely agreed with human experts at all; after several rounds of refinement it agreed with them nearly as often as the experts agreed with each other. That ceiling is the right target, because no grader can be more consistent than the people defining what good looks like.

5. Score the steps, not only the result

For any workflow with several steps, measure each step as well as the final output. It tells you where things go wrong, and it catches the run that reached the right answer by a route it should not have taken: a record it was not allowed to read, an action nobody approved. A correct answer reached the wrong way is a failure in an enterprise, even when the final output looks fine.

6. Run every case more than once

AI systems give different answers to the same input on different runs. A case that passes once may fail on the next attempt. Run each case several times and report two numbers: how often a single run succeeds, and how often every run succeeds. The second is what a customer or an auditor experiences.

The two can point to different models. On one public test of banking, consulting and legal work, the model with the best single-run score was not the one that succeeded most consistently across four attempts. For a workflow that runs thousands of times a day, consistency usually matters more than the occasional brilliant answer.

7. Put quality, cost and speed on one scorecard

Quality alone does not decide anything. A model that is two points better at forty times the cost, or twice as slow in a live conversation, may be the wrong choice. Put every candidate on one scorecard: success on the business outcome, consistency across runs, critical errors, cost per accepted outcome and response time.

Then price the errors. Not every mistake costs the same. In an expense audit, a misspelled merchant name rarely changes a decision, while a wrong total can trigger or miss an audit worth real money. Multiply each error type's rate by its cost, and the team will spend its time on the errors that matter rather than the ones that are easiest to count.

8. Set the thresholds before you look

Decide what a workflow must achieve before it ships, and write it down before the results come in: the minimum success rate, the maximum rate of critical errors, the cost and response-time ceilings. Thresholds set after seeing the numbers have a way of landing just below them.

A published example shows how this works in practice. A contact-center question-answering service handling about 5.3 million chats a month had to replace its model when the provider retired it. The team defined its gates in advance: the output had to follow a required format, correctness could not fall, the rate of "I don't know" answers and the response time could not rise, and the style had to stay within limits. Calibrated against 146 human judgments, those gates eliminated candidate after candidate: two could not follow the format, three could not show they were at least as accurate, one was too slow and one broke the style rules. The model that passed was nearly 10 points more accurate than the one it replaced and answered in a little over half the time.

9. Treat every change as a release

Re-run the full set whenever anything changes: the model, the prompt, the data the system retrieves from, the tools it can use, the scope of what it may do. Model changes deserve the most attention, because they now arrive every few months whether you want them or not, and because a newer model can regress on your work while improving everywhere else.

A well-kept test set turns a forced migration from a risk into a routine. Notion, for example, has been able to put new models into its product the day after release, because the measurement was already in place to show whether they were better.

10. Keep measuring in production

Passing tests is not the same as working in production. In a mid-2026 survey of 157 enterprises, about half had shipped an AI feature that passed internal evaluation and then failed in front of a customer. The most common reason given was that the tests did not reflect real-world conditions.

So the measurement continues after launch. Roll out in stages, comparing the new version against the current one on live outcomes before switching everyone over. Sample live work every week and grade it the same way as the test set. And track a small number of honest production metrics:

• Resolution, not deflection. Did the issue get solved, and did the customer come back about it?
• Accepted without changes. How often do people use the output as it is? An independent study of more than 23,000 AI-drafted clinical notes found that 84 percent were edited before signing, and that the individual clinician predicted the amount of editing better than the specialty did. Compare each person to their own baseline.
• Disagreement and uncertainty. For an agent that makes decisions, how often does it disagree with the person who reviews it, and how often does it say it is not sure? Ramp tracks exactly these two for its policy agent, because pushing one down tends to push the other up.
• Escalations and reversals. How often does a person have to step in or undo what the system did?

Every failure found in production becomes a new test case. That loop, from tests to staged release to live measurement and back to tests, is what keeps a workflow good as everything around it changes.

How the platforms build this in

The enterprise platforms that take this seriously have converged on the same shape. Palantir's evaluation tooling generates test cases from a company's real business records, grades each step of a multi-step workflow, runs cases repeatedly to expose variation, compares models and prompts side by side on quality and cost, and lets experts review results in simple applications rather than spreadsheets. Agents that change records are tested against a simulated copy of the business, so a test never touches real data. Its chief technology officer has put the principle well: the goal is "not one of benchmaxing, but of benchmaking." The major cloud platforms now ship evaluation services along similar lines, with graders, sampling of live traffic and regression checks before release.

The tooling matters less than the discipline. A team that reads its own cases, defines success honestly and re-tests on every change will outperform a team with better tools and no habit of using them.

Where Fig fits

Fig builds this measurement into every deployment rather than selling it separately. Our forward-deployed engineers start with the workflow as it runs today, so there is a baseline to measure against. They collect real cases with the people who do the work, write the edge cases with the people who own the process, and calibrate any automated grader against those same experts. Every correction a reviewer makes in Command Center becomes a labeled example, so the test set grows as a side effect of the work.

Because we are not tied to any model provider, the same measurement decides which model handles each workflow, and re-decides it when a new model arrives or an old one is retired. That is what lets us route each task to the model that is actually best for it on your work, at the lowest cost that clears the bar. And the test sets, graders, thresholds and results belong to the customer, in a form their own engineers can read and run.

Key takeaways

• Your benchmark is not your business. Real enterprise work scores far below public tests, and the best model changes from task to task.
• Measure a workflow, starting from real cases. Read a hundred of them, write down what goes wrong and measure what fails most or costs most.
• Define success as the business outcome, graded with the simplest check that works and AI graders calibrated against your experts.
• Run everything more than once, and put quality, consistency, cost and speed on one scorecard with errors priced in dollars.
• Set thresholds before you look, and re-test on every change, especially model changes, which now arrive every few months.
• Keep measuring in production with honest metrics, and turn every live failure into a new test.

If you want to see how this would work for one of your workflows, talk to us.