What Frontier Labs Buy When They Buy AI Training Data

Handshake AI’s training business reportedly reached $1 billion in gross annualized revenue roughly a year after launch. ¹ That is a sizable reported revenue figure, but the source does not say which products generated it, who the customers were, or what the deal terms looked like. ¹ The interview describes expert-data work, evaluation design, and benchmarking rather than establishing a defined, priced catalog for buyers. ¹

The revenue signal is real; the product is still undefined

Latent Space reports the $1 billion figure and the roughly one-year timeline, but does not break the revenue down by product, customer, or deal size. ¹ The discussion covers finance, life sciences, coding, and alignment, and includes evaluating new datasets, testing training data on open models, and designing private enterprise benchmarks. ¹ It does not identify which activity, if any, accounts for a particular share of revenue or establish that each is available as a repeatable, priced product. ¹

Analysis: The procurement question is what a buyer would actually receive: licensed examples, expert-generated tasks, evaluation infrastructure, managed human work, or some combination. The source describes some of these areas, including expert data and evaluation environments, but does not provide a product catalog, revenue split, or comparison with competing vendors or earlier Handshake products. ¹ It is therefore a weak basis for claims about market share or relative quality.

What frontier labs buy: expert data, evaluations and benchmarks

Three examples in the interview show the range of work discussed: an agent handling a simulated bank inbox and producing a workbook for Project Falcon; VIALS, a benchmark for interpreting scientific images; and a coding simulation that lets evaluators inspect an agent’s process as well as its final response. ¹

Domain	Work described	What the example may reveal
Finance	In Project Falcon, an agent finds its assignment in a bank’s email, works out which files govern the deal, and delivers a workbook. 1	Whether a model can carry information through a multi-step task, rather than answer only a direct question. 1
Life sciences	VIALS presents scientific images, including gel and brain-tissue tasks. 1	How a model’s judgment on a defined visual task compares with an expert’s. 1
Coding	In a simulated file-access task, evaluators can inspect which files an agent read, whether it checked copies, and what it did to the shell history. 1	Process behavior that a transcript of the final response would miss. 1

The finance example highlights a specific failure: the agent can explain the tax treatment when asked directly, yet fail to recognize where the rule applies in the task. The source describes the agent carrying a number into Python and subtracting it, and identifies the mistake as failing to apply the rule to the relevant line of code. ¹ For a buyer, this suggests testing whether a system applies domain knowledge within a workflow, not just whether it can explain a rule. For another angle on evaluating finance agents, see FINCOSMOS Failed Every Session—That's the Finding.

Project Osprey is a separate published deal traced from a client email to a grading decision; the deal shown in the interview is Project Falcon. ¹ Keep the two examples distinct when assessing what the interview demonstrates.

Analysis: The finance example supports testing end-to-end task performance, not explanation quality in isolation. A benchmark that scores only standalone answers could miss failures that occur when a model must locate a source, apply a rule, and produce a deliverable.

What the benchmarks measure — and what they don’t

VIALS is described as a benchmark for interpreting scientific images. In the brain-tissue example, Nick Magazine asks which two slides show the most advanced vessel necrosis, selects H and I, and the model selects H, I, and G. ¹ That single disagreement makes the evaluation concrete, but it does not establish an overall error rate.

The coding simulation provides a different kind of evidence: evaluators can inspect which files an agent read, whether it checked copies, and what it did to the shell history. ¹ The source also discusses evaluating new datasets, testing training data on open models, the limits of rubrics compared with expert judgment, and private enterprise benchmarks. ¹ It does not establish that any of these approaches predicts performance on a buyer’s own tasks. ¹

Analysis: A benchmark measures performance on the task it defines; it does not by itself show that results transfer to a customer’s workflow or that training on related data caused an improvement. The source gives one VIALS disagreement, not an error rate or an independent validation result, and reports no causal improvement tied to the training data discussed. ¹ Ask what is scored, what counts as correct, and how closely the evaluation resembles the work you need done.

For expert-led tasks, ask where human judgment enters: task creation, grading, validation, or escalation. The source discusses the limits of rubrics compared with expert judgment, but does not describe the operating process behind the expert data. ¹ Analysis: Ask how disagreements are handled and what evidence supports the final score; the right balance of rubric-based grading and expert review depends on the task.

AI data licensing and training data provenance: what the source leaves out

The source does not specify rights, licensing terms, or provenance documentation for the training data discussed. ¹ A benchmark result or expert judgment, by itself, does not tell a buyer what material went into a deliverable or how the buyer may use it. Different stakeholders need different documentation, a theme explored in AI Transparency Isn't One Document — It's Several, for Different Readers.

Diligence to request before signing:

• Identify the inputs. Which items are customer-provided material, source documents, generated tasks, expert judgments, or evaluation artifacts, and how is each represented in the deliverable?
• Map rights to use cases. Can the material be used for training, evaluation, internal development, and commercial deployment? Can it be retained, shared with affiliates, or redistributed? The source states none of these terms. ¹
• Ask for traceability. Request provenance records for each deliverable, or a defined sampling and audit method. Ask how an expert judgment is linked to its task and source item.
• Understand expert review. Ask how experts are recruited, instructed, and reviewed; how judgments are recorded; and how disagreements are resolved. The source discusses expert judgment but does not describe this operating process. ¹

Keep rights to data separate from rights to evaluation results in the contract. What a buyer can do with training examples and what it can do with benchmark scores are separate questions, and the source resolves neither. ¹

How to test AI training data vendors before you sign

Start with the failure mode in the finance example: test whether a system applies domain knowledge inside a full workflow, not just whether it can explain a rule on request. ¹ Score whether it finds the controlling document and produces the required artifact, rather than grading only an isolated answer.

Match the evidence you request to the work:

• Scientific data: Request task definitions, grading and adjudication procedures, and hard cases. Ask how the vendor handles disagreements like the VIALS slide selection and whether independent validation exists. The source provides one disagreement, not an error rate. ¹
• Coding: Ask whether deliverables capture process behavior—such as file access, copy-checking, and shell-history actions—or only final code. ¹
• Training claims: Separate the quality of a training set from the quality of the benchmark used to assess it. The source mentions testing training data on open models but reports no causal improvement result. ¹

Analysis: Where rights allow, pilot against your own holdout or private benchmark. Agree on tasks, scoring, and acceptance criteria in advance, then check whether results reproduce on your task distribution. A strong score on a vendor’s benchmark is context, not validation on your workload.

LLM training data procurement: vendor strategy and contracts

The reported revenue figure indicates substantial activity in Handshake AI’s training business, but the source discloses nothing about revenue mix, customer concentration, or product-level economics. ¹ It cannot establish whether the business rests on reusable data, managed expert work, evaluation infrastructure, private benchmarks, or a combination. ¹

Analysis: Before committing to a provider for specialized work, specify what you are buying and what happens if the relationship or deliverable changes. Contracts can address access to data and provenance records, permitted reuse, retention, transition support, and the ability to validate future releases. They can also define change control for task definitions, graders, source material, and benchmark versions—subjects the source does not address. ¹ Validation can become a buying decision of its own; AWS Continuum's Two-Part Pricing Turns Exploit Validation Into a Procurement Problem takes up a similar theme in another setting.

The reported revenue is evidence of a sizable business in specialized training work, but it does not establish that any particular vendor’s deliverable is rights-clear, traceable, or effective on your workflow. ¹ Before signing, ask for the rights documentation and quality evidence relevant to the deliverable, not just a revenue figure.

Source

¹ “What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team,” Latent Space