# How a model learns a job
A layer-by-layer playbook for building an AI model that knows your company's work: compute, base models, data, evaluation, training, serving, and the three ways to put them together.
- Published: 2026-11-10
- Type: Playbooks
- Topics: sovereign-ai, model-choice, agents
- Canonical: https://hellofig.io/blog/how-a-model-learns-a-job
**In short.** A model that knows your work is built in layers: compute, an open base model, your data, an evaluation set, training, serving and the application on top. Each layer is now a market. Companies can buy a vertical app, build on a platform inside their own environment, or assemble the pieces, and each route gives them different ownership.
---
On Oct 31, every evaluation a company had built on OpenAI's platform became read-only. On Nov 30, OpenAI [shuts down](https://developers.openai.com/api/docs/deprecations) its Evals dashboard and API, along with Agent Builder, its tool for assembling agents, and points customers to Promptfoo, an outside testing tool, instead. OpenAI is also [winding down](/blog/everyone-wants-their-own-model) self-serve fine-tuning, the service many companies used to train their first custom model.
The same week Evals went read-only, Snowflake [launched a preview of Cortex Training](https://snowflake.com/en/blog/cortex-training-fine-tune-open-weight-llms), which trains open models inside a customer's own Snowflake account. Fireworks, Applied Compute, Trajectory and Thinking Machines all sell training as a service.
The work of building a custom model is moving from the big labs to a market of specialized vendors, and each layer of a trained model now has its own suppliers. This third post in our series is a guide to those layers, using the companies from [the previous post](/blog/everyone-wants-their-own-model) as examples.
## Before you train
Training usually comes third, after prompts and context, and the order runs from cheapest to most expensive. Changing the prompt takes minutes. Giving the model the right documents and data at the right moment takes hours or days. Training takes longer, costs more and has to be repeated when a better base model comes out.
Most training vendors give the same advice, and we agree with it: get as far as possible with the context and the agent's design first, then train the model to use its tools better. Trajectory draws a useful line between facts and judgment. A fact that changes, such as which companies have been delisted, belongs in the context the model reads. Judgment, such as how this firm decides which fraud alerts to escalate, belongs in the weights.
A task is ready for training when it runs at high volume, where a cheaper model saves the most, or when better prompts and context have stopped moving its score on the company's own tests.
## What training adds
Every model starts with pretraining: months of reading a large share of the internet to learn language, facts and reasoning. Only a few labs and a few dozen neolabs do this. Everyone else starts from a pretrained model and trains it further, which is called post-training, usually in one of three ways.
Supervised fine-tuning shows the model examples of the job done well, such as a thousand fraud cases with the right call on each, until it imitates them. It is the cheapest and fastest method and works well for narrow, repetitive tasks, especially where the model's format or behavior is wrong more than its knowledge. It works like learning from a textbook of right answers.
Preference tuning shows the model two answers and which one a person preferred, so it learns what good looks like when there's no single right answer.
Reinforcement learning lets the model attempt a task many times and scores each attempt, through a test that passes or fails, a rubric written by experts, or a grading model. The model learns from whatever scored well. This is how labs taught models to reason, which we covered in [When AI can do more than we can check](/blog/ai-agent-teams), and it's the method behind most of 2026's strongest custom models. It is the closest of the three to how a person becomes good at a job, by doing it many times and getting feedback.
Two techniques make training cheaper. Distillation trains a small model to copy a large one on a specific task, so the large model's skill can run at the small model's cost. Sierra makes the case that a company doesn't need its most expensive model to classify the purpose of a phone call, and most of the model calls inside its conversations run on small fine-tuned models. LoRA, a technique for training a small add-on to a model and leaving the original weights untouched, makes training fast enough to repeat often and lets one base model carry many task-specific add-ons.
## Compute
Training a narrow model now costs very little. Coinbase trained its fraud model for under $100 on one GPU. NVIDIA and Palantir [trained](https://developer.nvidia.com/blog/from-wafer-out-to-first-token-codifying-supply-chain-expertise-with-nemotron-and-palantir-foundry/) a supply-chain planning model in minutes on two of NVIDIA's B200 chips, using LoRA.
Harvey's legal model took far more: [Tenet](https://harvey.ai/blog/post-training-update-harvey-tenet) used about 150 of NVIDIA's newest B300 chips for about two months.
Most companies rent this capacity from a cloud or one of the newer GPU clouds such as CoreWeave, Lambda and Nebius. A growing number run it on their own hardware. Revolut serves its coding model from 56 B200s it controls, and Palantir runs training inside customers' own data centers, including ones with no internet connection.
## The base model
The base model limits how good the trained model can get. Most companies that train now start from an open model, partly because the closed labs are pulling back from customization: a model customized on a closed model is retired when the original is, as the [previous post](/blog/everyone-wants-their-own-model) describes. AWS still lets customers train its own Nova models through Nova Forge, but the result stays inside AWS.
With an open model, the company holds the weights, the trained model file itself, so no one can retire it. The most used are Chinese: Moonshot's Kimi, Zhipu's GLM, Alibaba's Qwen and DeepSeek. American options grew this year. NVIDIA's Nemotron family comes with its training data and recipes, Thinking Machines released Inkling in July, and Google's Gemma runs in air-gapped data centers, ones with no connection to the internet. Harvey built on Kimi, NVIDIA and Palantir on Nemotron, Airbnb on Gemma and Coinbase on Qwen.
A company choosing a base should check four things: size (smaller models are cheaper to train and serve), license terms above a revenue threshold, country of origin and any rules on it in its industry, and how often the maker ships improved versions it can retrain on.
## Data
Data is where companies differ most.
The four places training data comes from, from most general to most specific to one company.
Expert-made data comes from professionals paid to write examples, grading rubrics and practice tasks. Mercor is the largest supplier. Its CEO said in August that it paid experts around $150 an hour and logged 2.5 million expert hours in a single quarter. Harvey used Mercor's lawyers for Tenet. This data teaches a model a profession, and anyone who pays can buy the same kind.
Synthetic data is generated by models, usually from a few real examples, and then filtered. NVIDIA uses its own tools to generate much of the data behind Nemotron, and says synthetic data still needs people to ground it, curate it and check the results.
Historical work is what a company already has: Coinbase's fraud outcomes, Airbnb's past pricing-rule violations, Intercom's billions of support conversations, a security firm's years of labeled code findings. This is usually where a first model comes from.
Client data adds a constraint. Harvey doesn't train on its law firm clients' documents at all. To test a model on due diligence, it plants known problems, such as contracts that don't tie together or a missing agreement, builds a mock data room of up to about 10,000 documents around them, and grades whether the model finds what was planted. Once firms see a model work this way, they often ask to run the same test on their own data.
Production records are the most valuable and the least common. Palantir's approach is the clearest example. When a planner makes a decision, its software records what was decided, why, what the person expected and what happened. When an AI recommends something, it records whether the person accepted, edited or overrode it. That record becomes training data, and a model can be tested against what happened in the past before anyone relies on it.
Anyone who pays can buy expert or synthetic data, and it teaches a model what any professional knows. Only the company has its own records, which teach a model how that company works.
## Practice environments
Reinforcement learning needs somewhere for the model to practice. The industry calls it an environment, and Mercor and others now sell them.
Mercor's chief executive, Brendan Foody, described the parts in an August talk. There is a world, such as the emails, documents and spreadsheets from a real legal deal. There are working copies of the applications a professional would use, such as Salesforce or Microsoft 365. And there are tasks, each with a way to grade the result, usually a rubric written by an expert the way a professor writes one for an essay. Grading is hardest outside math and code, where a model can't reliably tell where it went wrong, and a good rubric has to anticipate the many acceptable ways to do the task and the dozens of ways to get it wrong.
Prices vary widely. Foody said Mercor's tasks range from about $50 to $10,000 each, with about $2,000 typical for custom work. In one example he described, Mercor built about 1,800 corporate-law tasks and spent roughly $500,000 on computing to train an open model on them, which raised its score on those tasks from about 5% to about 27%. These are the company's own numbers, on tasks hard enough that a model passing a quarter of them was a large step.
Before a run starts, check whether the base model can do the task at all: if it fails all of 16 attempts, it has nothing to learn from, and if it succeeds once or twice, training can build on that. Then check whether the grading can be gamed. In one case Fireworks has described, a model told to minimize compile errors learned to write no code at all. Models find the easiest path to a high score, so someone has to read what they produce as well as the scores.
## Evaluation
Yash Patil, Applied Compute's chief executive, describes reinforcement learning as a machine for climbing hills, where the hard part is defining the hill. Bridgewater set an 80% accuracy bar on its own tasks before it compared any model.
A mature evaluation set is a few hundred real cases with agreed answers or grading rules, kept away from the training data. It tells a company whether a trained model beats the frontier, whether a new frontier release got worse at its task, and whether the next version of its own model is an improvement. We covered how to build one in [How to measure applied AI](/blog/measuring-applied-ai). With OpenAI's Evals closing, it is also a reason to keep tests in tools a company controls.
The best evaluation sets come from real traffic, including requests the current system gets wrong, and are graded through the same software the model will run in. They don't need to be large to start. Palantir tells customers that ten tests tied to business outcomes are enough to begin, and they grow from there.
Companies disagree on whether to publish them. Harvey releases some of its legal datasets, on the theory that a benchmark gets better when many people train against it, and the labs then report scores on it. Applied Compute advises the opposite, because a public benchmark becomes a target that models are tuned for. Our read: keep the tests that decide what ships private. Publish only those whose purpose is to get the labs to improve at your field.
## Training
Companies can get training done in five ways.
- Do it in-house with open-source tools such as NVIDIA's NeMo and Hugging Face's libraries. This takes machine learning engineers, who are scarce and expensive.
- Use a training API. Thinking Machines' Tinker runs the computing so a team only writes the training logic. Bridgewater and Glean used it.
- Hire a research partner. Applied Compute embeds researchers with customers, and Fireworks ran the reinforcement learning infrastructure for Harvey's Tenet.
- Use a self-serve product. Trajectory connects to a company's applications, collects feedback on how the model performs and retrains it.
- Train inside a platform the company already uses. Palantir, Snowflake, Databricks and AWS's Nova Forge all now offer this.
In Menlo Ventures' [Nov 3 report](https://menlovc.com/wp-content/uploads/2026/11/menlo_ventures_enterprise_ai_report-2026-110326.pdf), 91% of companies that post-train use an outside partner. The companies that already monitor software in production are buying their way in, because they already hold the records training needs. Datadog [bought Adaptive ML](https://www.datadoghq.com/about/latest-news/press-releases/datadog-acquires-adaptive-ml-to-accelerate-its-investment-in-ai-research-and-development/) in June to train models on the signals its customers already collect, and Dynatrace [bought Arize](https://www.techtarget.com/it-strategy/news/366648773/Dynatrace-acquires-Arize-for-AI-agent-development), an AI evaluation company, for $915 million.
## Serving
A trained model still has to run, quickly and cheaply, every time someone uses it, which the industry calls inference.
Fireworks, which hosts open models for companies, [passed](https://fireworks.ai/blog/series-d-announcement) $1 billion in annualized revenue in July at a $17.5 billion valuation. It says more than 95% of the tokens it serves come from models customized on customers' own data. Applied Compute says 40% of its customers buy both training and serving from it, and its pitch is that whoever trains a company's model usually ends up serving it.
The alternative is running the model on your own chips, as Revolut does. Airbnb's numbers show when that pays: its small model cost about 1% of a frontier model at thousands of requests a minute, but below about 30 requests a minute a frontier model was cheaper than a dedicated chip.
Speed is often as important as cost. Coinbase's fraud model answered in about 0.7 seconds against 1.5 for the frontier model it replaced, and Airbnb's classifier answered in under 85 milliseconds even at its slowest, where frontier models typically took about 900.
Before switching over, run the evaluation set through the production setup as well as the training setup. Training and serving software can do the math slightly differently, and quality can drop without anyone noticing.
Most companies that train their own model end up routing. Each request goes to the company's own model when it can handle the task, and to a frontier model when it can't. Clay cut its inference costs by up to 47% with a router Applied Compute trained.
## The application on top
The last layer is what people use: the agent, its tools and permissions, and the screens or chat where work happens. This layer carries the company's workflow, and much of it will change as models improve. In [The model is eating the harness](/blog/the-model-is-eating-the-harness), we argued that the parts worth investing in are permissions, audit, data connections and cost controls, because models absorb the rest.
## The team
A first model takes a small team. Harvey's data research team was seven people as of August. We think a company's product team, the people who know what a good answer looks like, are the best judges of training data, and that product and machine learning teams will end up working as one. Pereyra said Harvey's early mistake was trying to hire lab researchers commanding pay packages of $100 million or more. Partners like Fireworks and Applied Compute let it work with a smaller team, because it no longer had to build training infrastructure itself.
For most companies, a first project needs one person who owns the task and its results, one or two engineers who can run training through a service, and the subject experts who write and check the grading rules.
## A first project, step by step
Here is how we would run a first project over about three months.
1. Weeks 1 and 2: pick one task that runs at high volume, can be checked and doesn't change every week. Pull a few hundred past cases with known outcomes, agree on how to grade them, and measure today's frontier model against them. This is the baseline.
2. Weeks 3 and 4: improve prompts, context and the agent's design, and re-measure. Many projects stop here because the gap closes. If it doesn't, the remaining gap tells you what training has to fix.
3. Weeks 5 to 8: choose a base model and check that it can sometimes succeed at the task. Fine-tune it on past cases, then use reinforcement learning against the grading rules if the task needs judgment. Read a sample of outputs after every run to catch gaming. For a first project, the practice environment can be as simple as the past cases and grading rules from step 1. Simulated workplaces with copies of real applications come later, for agents that work across many tools.
4. Weeks 9 and 10: run the trained model alongside the current system on live work without acting on its answers, and compare both on the evaluation set and on real outcomes.
5. Weeks 11 and 12: route a share of live traffic to the trained model, with a rule for sending it back to the frontier model if its scores drop. Start recording every correction people make, because those records are the data for the next round.
For a narrow task, the computing bill for the whole project can be hundreds or a few thousand dollars. Most of the cost is people's time spent on data and grading.
## Three ways to put it together
Companies get a model that knows their work in one of three ways, and each gives them different ownership.
Three routes to a model that knows your work, and who ends up owning what.
The first is to buy an application built for your profession, the way law firms buy Harvey. The vendor trains on what lawyers know, collects expert data across its customers and runs the model. A firm gets a strong model quickly. It owns the work product but not the model, and the profession's knowledge is shared with every other customer.
The second is to build on a platform that runs inside your own environment, the Palantir route. The platform connects the company's systems, records its decisions and trains open models on them. The company keeps the data, the evaluations and the weights, and the model reflects how this company operates.
The third is to assemble the pieces: a training API or research partner, your own evaluation set, and a serving provider or your own chips. This gives the most control and the most flexibility on cost, and it needs a team that can run it. Coinbase and Bridgewater took versions of this route.
Many companies will use more than one. A bank might buy a vertical app for legal review, build on a platform for credit operations and assemble its own fraud model.
## Choosing a route
1. Is the edge in the profession or in the company? If a task looks the same at every firm, a vertical app trained across many firms will usually beat anything one company can train. If the edge comes from how this company decides, its own records are the training data, and a platform or assembled route keeps them in-house.
2. Where must the data stay? Regulated data and national security work often rule out a vendor's cloud. That points to a platform that runs on the company's hardware.
3. Does the company have people to run it? Assembling the pieces needs machine learning engineers who can debug training runs and maintain evaluations. Without them, a platform or a research partner fills the gap.
4. Who owns the weights at the end? Ask every vendor whether the trained model can be exported and run elsewhere, and get the answer in the contract.
5. What happens when a better base model comes out? A company that keeps its data and evaluations can retrain on the new base in days. One that doesn't starts over.
Owning a model makes sense one task at a time, and the next post looks at what keeps it improving after it ships. In [AI systems that improve themselves](/blog/ai-systems-that-improve-themselves), we follow the loops that turn production records into a better model, a better agent and a better workflow, and what that looks like in 2027.
