Legora Says It Can't Read Client Traces—What Counts as Proof a Legal AI Agent Works?

Legora CTO Jacob Lauritzen says the company cannot read its agents' production traces. Almost everything users upload is confidential, and much of it belongs to the law firm's clients rather than the firm.¹² Legora describes other ways of evaluating and monitoring its agents, including simulated tasks, task-level results, failure categories, and checks in production.¹² Those methods can provide useful signals, but the interviews do not explain how well the tests predict performance in a particular firm's deployment.

Replacing a benchmark score with a live matter would not, by itself, resolve that question. Legora describes simulated customer work and production checks as parts of its approach to evaluation when it cannot read traces.¹² Buyers still need to understand what each method measures and what it leaves untested.

Why can't legal AI vendors inspect agent traces?

Lauritzen says Legora cannot read its agents' production traces because uploaded documents are confidential and much of the data belongs to the law firm's clients.¹² That restriction also limits the company's access to the agents' work on real matters. Analysis: This can make diagnosing specific production failures more difficult, though the interviews do not quantify that effect.¹²

For a buyer, the practical question is what evidence a vendor can provide without exposing client material, and what each piece of evidence actually measures.

Analysis: It helps to separate the evidence into three layers: simulated evaluations before deployment, privacy-conscious signals from production, and validation by the firm on its own tasks. This is a buyer-oriented framework based on methods Legora describes; the interviews do not say that any one layer is sufficient.¹²

What can legal AI vendors measure without exposing client matters?

Legora's evaluation team pairs three engineers with three lawyers to build test environments modeled on customer work. The interview compares these environments to those used in reinforcement learning.¹² One exercise places an agent in a 10,000-file data room, asks it to conduct due diligence, and requires a report that identifies the relevant red flags.² For another take on testing agents in interactive settings, see CogArena's interactive behavioral experiments.

That exercise tests finding relevant material, identifying issues, and producing a report. It does not, on its own, show how the same agent performs on a firm's live matters. A simulation is evidence about the task it represents, not a guarantee for every deployment.

Legora also breaks results down by practice area and task. The interview says a new model can improve some kinds of legal work while performing worse on others.² That is more informative than a single blended score, which could conceal a regression in a task a firm cares about. The same logic is behind the argument for behavioral diagnostics rather than aggregate scores.

Analysis: Buyers can ask for the same breakdown, along with the document conditions and expected output format. A firm assessing a research workflow needs evidence about research, not just an average across unrelated tasks.

Evidence layer	What it can show	What firms still need to check
Simulated evaluation	How an agent performs on a defined task in a controlled environment, such as the 10,000-file data-room exercise.2	How the environment and expected answers were built, and how closely they represent the firm's own work.
Production checks	Legora says it classifies production failures into known types and runs language-model checks behind feature flags.2	What the checks assess, how they are validated, and what information they use or retain.
Firm-specific validation	Analysis: Whether evidence applies to the firm's own tasks and deployment conditions. The interviews do not describe this as a Legora evaluation method.12	Whether the firm can review or independently verify results before relying on them.

What do law firm AI evaluations show, and what do they leave out?

The 10,000-file exercise is a concrete task description, not a validation study. The interviews do not explain how representative the simulated data room is of real matters, how expected answers were created, who reviewed them, or how often the test is refreshed.¹² They also do not connect simulation results to outcomes in a particular firm's deployment.¹²

These gaps do not show that the methods fail. They do limit what a score can support: a test may track changes on a defined task without establishing broader readiness.

Analysis: A useful evaluation report would let buyers inspect what was tested and how. A diligence checklist could include the task definition, test-set composition, ground truth, reviewer qualifications, error categories, test conditions, and how the test set changes over time. The interviews do not provide those details, so this is a buyer checklist rather than a description of Legora's process.¹²

How can production performance be monitored when traces stay restricted?

Legora says it classifies production failures into 10 to 20 known modes and runs low-cost language-model checks behind feature flags.² A companion interview describes paraphrasing traces and narrowing failures to 10 or 20 types in the context of tracking product metrics without reading traces directly.¹

Those practices may help identify recurring patterns without routinely inspecting client material. But the interviews do not explain how paraphrasing works, what information remains in the resulting traces, or precisely what the production checks assess.¹² They also do not report the checks' accuracy or false-positive and false-negative rates.²

Buyers can ask for the failure taxonomy and definitions of its categories, then ask how the checks are validated. Can the firm see results for its own tasks? How are ambiguous cases resolved? What information is exposed or retained? The interviews raise these questions but do not answer them.¹²

The interview account also describes a citation result that moved from 70% to 99% after one release, without Legora needing to change anything.¹² The excerpts do not explain why the result changed. Analysis: That makes the measurement method and the release conditions important context when interpreting a reported score.