Legal AI benchmarks score an agent once. In practice, a legal agent is scored many times: a lawyer reads its work, says what is wrong, and someone changes the agent. A production agent goes through a dozen or more of those corrections. Each one can fix an error, and each one can teach a new one.

I call the record of an agent’s errors across those corrections its trajectory. This post shows how we measure a hallucination trajectory on real NDAs, what our first run found, and what we changed because of it. Not all of it went the way we expected.

We turned ContractNLI into a stream of corrections

ContractNLI (Koreeda and Manning, Findings of EMNLP 2021) has 607 real non-disclosure agreements collected from SEC filings and the web. Each is labeled on 17 standard statements, such as “Receiving Party may share some Confidential Information with some of Receiving Party’s employees”: does the contract say it (“yes”), say the opposite (“no”), or say neither (“not mentioned”)? The labels also mark the sentences that support each answer.

The benchmark scores one pass over each contract. We changed it into an on-the-job test:

  1. A Codex agent (gpt-6.1-sol) in a project folder gets one NDA per chat, answers the 17 questions, and gives a quote from the NDA for every “yes” and “no”.
  2. After each of 16 teaching NDAs, code compares its answers with the dataset’s labels and sends a correction in the same chat. The first correction, word for word:

    Question 6 (Sharing with third-parties) needs to be “no”, not “yes”, because the NDA says: “In the event that the Vendor requires the assistance of any third party other than employees of the Vendor, to whom disclosure of any Confidential Information is necessary, the Vendor shall first seek ISCA’s written approval of such third party …”

  3. Before the first correction, after 8 and after 16, a throwaway copy of the agent answers 24 NDAs it has never seen. That is 408 answers, scored by code against the labels.

We ran two agents side by side on the same NDAs and the same kind of corrections:

  • Plain Codex, told only “Keep my corrections in this project, so they apply next time.” It kept them as rules in its AGENTS.md file.
  • Codex + Method, our product. Each correction becomes a tested change to the agent’s workflow, and a code step checks every quote against the NDA before an answer is saved.

We looked for two kinds of hallucination

A lawyer who says an agent hallucinated can mean several things. We measured two:

  • Invented citations: a “yes” or “no” whose quote is not in the contract.
  • Learned wrong rules: an answer the agent got right before a correction and gets wrong after it, on contracts it has never seen.

The first can be checked on a single run. The second only shows up across a trajectory.

ContractNLI: Yuta Koreeda and Christopher D. Manning, “ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts”, Findings of EMNLP 2021, CC BY 4.0. stanfordnlp.github.io/contract-nli.