The hidden edge in legal AI: Seeing what the model misses

Newsletter #24

Read to the end for some canine info

A word from the person behind the laptop

Andrej Karpathy said something on the Dwarkesh Patel podcast that I’ve been thinking about for a while: “reinforcement learning is terrible”

Modern assistants are shaped by reinforcement learning, where long chains of output get judged with thin, end-of-process feedback. That nudges models toward answers that look right, not reasoning that’s actually strong. At scale, that can narrow legal research.

What does that mean for the way AI is currently being used for legal tasks?

That’s the theme of the newsletter today.

Subscribe now

Devil is in the detail

Lawyers are sticklers… mostly.

The “Suits” stereotype is obviously far from the truth, but one thing it gets right is that really good lawyers do share a kind of obsession with details.

Finding the right court decision to change the view of a well established precedent. A footnote that leads you astray and think diffrently about a problem. And so on. And then of course tell all of your lawyer pals about it afterwards.

With the implementation of AI systems in legal research, this comes under pressure.

We’re living throught the biggest shift in legaltech to date. Period. It doesn’t matter if you don’t like or believe in AI. Every firm I talk to is either testing or fully onboarding AI tools and that itself makes it the biggest change.

We’ve gone from simply being able to access legislation digitally across jurisdicitons (already a revolution at the time) to automating fairly complex workflows. And the promise keeps grwoing.

Tbh, I’m a big believer in the potential of AI in the legal sector. As an entrepreneur building AI tools for lawyers, I have to. But we also need to be clear-eyes about the potential challenges that comes with a new way of working.

Because the way reinforcement learning in AI products work has the potential to shape how the next generation of lawyers think and work.

And not always as expected.

If junior lawyers, paralegals and students start using the same handfuld of models for the same types of tasks (which, in my opinion, they will) there’s a real risk their understanding of the law diverges dramatically from the generation before them.

The same way our relationship to legal research is different and digital compared to our parents generation manually digging through legal sources. But this shift feels bigger somehow.

Now lawyers are also outsourcing pieces of their legal research and reasoning to a model that doesn’t actually reason (controversial statement, depends who you ask).

Short note on reinforcement learning

Reinforcement learning is bascially trial and error at scale. The model tries a bunch of possible answers. Most of them are unusable. A few are fine. The ones that happen to get rewarded, either by a human reviewer or an automated scoring system, are the ones the model leans into next time.

It’s a little like training a dog: good dog, bad dog.

This is the actual the description they’re using at Mathworks:

Using reinforcement learning terminology, the goal of learning in this case is to train the dog (agent) to complete a task within an environment, which includes the surroundings of the dog as well as the trainer.

First, the trainer issues a command or cue, which the dog observes (observation). The dog then responds by taking an action.

If the action is close to the desired behavior, the trainer will likely provide a reward, such as a food treat or a toy; otherwise, no reward will be provided.

At the beginning of training, the dog will likely take more random actions like rolling over when the command given is “sit,” as it is trying to associate specific observations with actions and rewards.

Unfortunately, the model can’t tell which parts of its “successful” answer were actually good. It doesn’t isolate the valuable reasoning. It doesn’t critique itself. It simply rewards the entire path that happened to lead to the answer including the detours, the irrelevant bits, and the lucky guesses.

“Sucking supervision through a straw”

In practice, RL “sucks supervision through a straw”. A painfully accurate description by Karpathy.

Basically a long chain of output gets judged by a thin end-of-process signal that up-weights everything in the winning path. Both good steps and lucky detours.

Here’s an example: Reviewers at a procurement practice, like most laywers busy (but well-intentioned of course), tend to reward memos that cite three leading cases. The model internalizes that vibe.

When a junior asks about invoking the “extreme urgency” exception after a strike-related supply shock, the assistant produces a clean outline with those familiar citations and frames necessity narrowly.

It never surfaces an older appellate decision that analogizes a comparable disruption. The weird outlier precedent that could flip the analysis. Nothing is obviously wrong. It’s just predictably narrow because those rewarded patterns dominate. And it’s perhaps a little bit lazy too.

Humans don’t learn (or conduct legal research) like this.

If you went down three wrong paths and one right one, you wouldn’t reward all four equally. Hopefully you’ll reflect. You’ll think. Re-consider. Learn:

“Okay, that part was solid. That part was nonsense. Don’t do that again. Do more of this.”

But current LLMs have no such pause, no internal review meeting. They just upweight everything that happened to appear in the winning trajectory.

My worry is that once that rewarded pattern becomes the default assistant for thousands of juniors, we risk a generation of legal reasoning optimized for familiar vibes rather than discovery of outliers. The model got a gold medal once, so it keeps running the same lap.

That’s not how you win close races… sorry, cases.

Now, you might add that in law, consistency is justice. If AI nudges juniors toward the same settled authorities, we cut variance and avoid rookie mistakes.

But RL-nudged tools standardize on what was rewarded, not necessarily what’s right for your matter. In practice, that promotes majority readings, buries minority holdings that matter, and misses jurisdiction-specific wrinkles.

Intangible change

We’ve focused so much on hallucinations for the past few years that I’m starting to think we’re at Woodstock Festival. Hallucinations are tangible. You can, at least in theory, point at them.

This case doesn’t exist. This quote is made up. You might be too busy or too tired to catch them, but the error lives in the output. It’s checkable.

What worries me more is the thing we don’t see, because it doesn’t show up as a clearly wrong answer.

The bigger risk is the slow, intangible change in how legal research is done when LLMs, trained with the same reinforcement learning patterns, keep reaching the same kinds of conclusions in the same kinds of ways.

Not obviously false, not obviously hallucinated. Just consistently nudging lawyers toward a narrow band of authorities, arguments, and framings that happened to be rewarded during training.

Of course its not only reinforcement learning that causes homogenization. Pretraining data + instruction tuning also push models toward “safe, average” answers. Reinforcement learning then amplifies whatever humans/metrics reward, but the bottom line is the same:

The search space of legal reasoning shrinks and you rarely notice what’s missing, because the model is likely to give you something plausible.

You don’t see the argument it could have made if it had been trained differently. You don’t see the weird outlier that never gets surfaced because it wasn’t in the slice of data that was heavily rewarded. The good lawyers will still of course be able to perform proper research (especially in places where time and money is not an issue), but expectations will change on the time spent on these tasks now that AI systems can support.

But a lot of lawyers don’t have the luxury of no constraints.

So what happens in practice is that “research” slowly gets stale, narrow and more predictable over time.

So what can we do about it?

Last point from the Karpathy interview, something people in ML circles have been talking about for years:

Why don’t we train models on the process, rather than the final answer?

This idea process-based supervision has been sitting in the collective imagination of ML researchers for years. In theory, it solves the whole problem. Humans could review the chain of thought step-by-step, marking which steps were actually good reasoning and which were nonsense. The model wouldn’t have to guess which parts mattered; we’d tell it.

So why hasn’t this saved us yet?

Three reasons:

1. Humans can’t reliably evaluate long chains of machine reasoning

2. The process the model produces is not guaranteed to be the actual process

3. It’s expensive and slow

So here’s my rather average advice:

1. Treat the AI output as a baseline. Never the endpoint

Great for speed, but needs work for the final judgement. Treat it as such.

2. Keep the manual research muscle alive

Even if tools get “really good,” the ability to find the weird case, the uncomfortable exception, or the overlooked angle will always have value.

Nothing I’ve seen suggests this skill is about to become unfashionable.

3. Teach juniors how these models actually learn

When I teach AI to lawyers or law students, we spend quite a lot of time on the technical aspects. Not just “here’s a good prompt.” Also statistics, reinforcement learning, Python etc. Some hate it, and some love it.

I fundamentally believe that having a basic understanding of the tech gives you much better tools to discuss and understand what’s going on under the hood.

My advice is to teach the underlying dynamic:

• RL reinforces patterns, not truth.

• Confidence is not correlated with correctness.

• “Legal analysis" in AI often just means “rewarded analysis.”

Use the assistant for speed. Use your judgment to widen the search. Consistency is the floor, not the ceiling.

The best lawyers will be the ones who notice what the model never shows them.