AI Field guide 1: On trust cost and senior squeeze

Newsletter #28

Read to the end to see if you feel bad for the senior lawyers

A word from the person behind the laptop

Even people that work outside the legaltech and AI domain can see what is happening right now. Legal has been one of the first areas with work related mass adoption, and that means we have a chance to tell some stories fromt the frontlines.

I was lucky enough to be invited by Tech Hub Aarhus to sum up my 10 years in legaltech with a focus on the new challenges and opportunities in legaltech/AI.

Here’s the first part of the field guide about some of the current challenges.

Next week I’ll follow up with some potential solutions.

Is a lawyers job mainly text creation?

This is a data protection impact asessment. A type of risk assessment carried out every day by compliance teams all over the world.

After working as data protection lawyer in tech, I conducted a lot of these and the process appeared highly automatable: it was structured, repetitive and governed by relatively defined legal requirements.

So I started my own company and started selling AI software to lawyers

The product did not become the booming business I originally expected, but building a tech product gave me access to something even more valuable: talking to lawyer after lawyers about AI, gave me a close view of how law firms, in-house teams and compliance functions are actually using AI today.

Right now, a lot of the conversation is still dominated by teenagers from Silicon Valley saying “This is so over for lawyers!” every time a new model drops, but there’s an inherent misunderstanding in their approach to legal work. It’s build on the assumption that the work is the text in the contract itself.

For the example of the DPIA - or data protection impact assessment:

We do not yet know whether the model was given the right facts.

How is data being processed, where it is stored and which sub-processors are involved.

We do not know whether the agreement reflects the way the parties will actually operate.

There’s a possibility that nobody has spoken to the client - or the venor. Nobody has evaluated the commercial risk. And, at this point, nobody has put their name on the document.

And so on…

So while the drafting time has been minimised, a lot of the assignment has stayed unchanged.

That difference is the subject of today.

1. What work actually disappears when AI produces the draft?

2. What work remains, and what work is created?

3. Who ultimately captures the value?

AI makes the visible output appear almost immediately. That creates a feeling of productivity and a very attractive business case.

The first trap is therefore mistaking the generation of output for the completion of an assignment. Secondly, once producing and editing text becomes cheap, we may produce much more of it actually generating MORE work.

Identifying the productivity trap

We’ve seen this before.

This is a graph showing the pages the Standard Federal Tax Report. It contains the tax code, regulations, rulings, cases and much more. In other words: legal content.

It illustrates an important possibility.

When word processing made it cheaper to create text we did not necessarily use that efficiency to produce fewer documents. We produced more text and more versions (FINAL_VERSION.3 EDITED_FINAL]. You know what I mean.

AI may enable the same dynamic to unfold again, but this time at a much greater speed.

I welcome you to the redlining armageddon.

Today, both parties use AI to review the contract.

The AI compares the document with the defined playbook and proposes a new limitation-of-liability clause.

The counterparty’s AI compares that clause with its playbook, rejects our changes and restores its preferred wording.

Everyone appears productive.

More clauses are reviewed. More comments are produced. Redlines are exchanged and “reviewed” by the lawyers (some people might want to call it spot checking).

Lots of contractual challenges are identified in the process, but they are challenges that wouldn’t have be mentioned under so-called normal circumstances (meaning pre ChatGPT 3.5) and the lawyers do not have the mental capacity to deal with thousands of redlines produced by a system (even though they might not want to admit that themselves).

Closing an agreement and potentially binding the company to an obligation with the counterparty usually requires judgment and decisions from people with authority. Commercial trade-offs are hard, but it becomes even harder when every. possible. issue. remains. open.

So while AI has reduced the cost of proposing a change, it has not necessarily reduced the cost of reaching an agreement.

Sometimes the result is merely inefficient, but when generated output moves through the organisation faster than it can be verified, the productivity trap becomes something more serious.

The danger is that the speed and fluency of the output (”oh, this looks about right”) allow an error to travel faster than the verification intended to catch it.

Case hallucinations are not a single problem

And sometimes we should probably verify once more.

A well-known Danish lawyer was caught red handed presenting ChatGPT infused documents to the Supreme Court of Denmark.

Invented court decisions have become a recurring problem for courts. We see stories in the media repeatedly, and they are often presented simply as an “AI problem”. In my view, that conclusion is incomplete.

Case hallucinations are not a single problem. A model may invent a case that does not exist. But a more difficult problem arises when the case exists and the generated summary misrepresents what it says.

For a busy lawyer, verifying whether a case exists is relatively straightforward. Verifying that a summary accurately reflects the facts, reasoning and holding of the case can be much more time-consuming.

This is not to sound too much like a tech-bro but: The problem is not the technology. In this case there is an obvious error in the way the technology was used within the production and review process.

So what went wrong between generating the draft with AI and then filing the document to the court?

It would be easy to dismiss this as a one-off problem involving a lawyer unfamiliar with the technology, but that’s not the whole story.

Market expectations are shifting day by day. External advisers (like lawyers) are increasingly expected to use AI where it improves efficiency.

This is happening at the same time as, we are moving from traditional deterministic software to probabilistic AI systems.

From deterministic to probabilistic software

Let us return to the Data Protection Impact Assessment (Yeah, I know).

With a traditional contract-automation tool, the client might complete a questionnaire. If data is transferred outside the EEA, the tool inserts a predetermined transfer clause. If sub-processors are used, it adds the clause approved for that situation.

The system can still produce an inappropriate legal work. The client may misunderstand a question, the template may be outdated or the underlying rules may be wrong so its not the same as “legally correct”.

But the process and system are both inspectable. We can see which questions were asked, which answers were given and which rule caused each clause to appear.

The same inputs should produce the same result and the assurance mainly comes from controlling inputs and rules:

An LLM works differently. If we ask it to draft a GDPR-compliant agreement, it generates the text it predicts will be appropriate. It may produce an excellent first draft even though most people know NOT to use AI for full contract templates today… we’ll leave that for another day. And as we know: where information is missing, it may fill the gap with an assumption.

That changes what we need to check. We need to evaluate 1. the substance of the generated answer as well as 2. the system producing it.

Very smart people are working very hard to create guardrails and steer the LLMs in the right direction, and there’s a lot of money to be made in this space, but as of now the probabilistic nature of the system means that verification moves downstream: from controlling specified rules in the software towards reviewing the substance of individual outputs.

How do we measure value?

When an AI business case is presented to a company today, it often looks something like this:

The productivity value might be the time saved producing a first draft of a legal assessment concerning the processing of personal data.

Technology and implementation costs include licenses, token usage, training and the non-billable time spent implementing the system.

If that was it, the calculation would the same as an ordinary software-procurement calculation. It changes when the software produces answers that are plausible rather than reproducibly determined by specified rules.

How do we measure value?

The conventional model measures the cost of producing an answer.

It does not yet measure the cost of being able to rely on that answer. That reliability is often assumed rather than priced explicitly in the AI business case.

How can we include it?

Measuring the trust cost

The missing element is what I call the trust cost.

Assurance cost is the additional cost of making AI-assisted work reliable enough to use compared with the existing human workflow. It includes verifying facts and sources, reviewing the substance of AI-assisted work, obtaining senior sign-off and ongoing evaluation of the system.

Residual expected loss is the estimated value of errors that may still survive those controls. It combines the probability of failure after the controls with the consequences if that failure occurs. Those consequences may include rework, client harm, liability and reputational damage.

Baseline: Incremental assurance cost is what we spend to reduce the risk. Residual expected loss is the expected value of the risk that remains.

When I was 14, I worked as a courier for a law firm.

One day, a document had to be delivered to a local court. During my route, I took a break and went home to eat. What I did not know was that the document had to be filed before 4 p.m. or the client risked losing the case. The immediate mistake was human, but the consequences would still have belonged to the law firm.

Anyway, it’s another way of saying that humans are not error-free either. For the comparison to be fair, the company must compare the AI-assisted workflow with the existing human workflow which can be hard because while humans and AI might both make mistakes, its usually not the same type of mistakes.

The point is that probabilistic production changes where assurance happens, and assurance can become a significant cost. So some of the burden moves downstream and onto the people that also have to take responsibility for the result, often the senior lawyers.

And those senior lawyers are already facing increased price pressure from clients.

Clients are adding pressure as well

Clients increasingly expect their external advisers to use AI and they do not expect the law firm to retain the entire efficiency gain.

Quote from: https://affarsjuridik.di.se/spotifys-toppjurist-juridiskt-korrekta-svar-racker-inte-alltid/

If work is billed by the hour, completing it faster may reduce revenue. I don’t think the billable hour is dead. The billable hour is just one way of measuring a service delivery.

Law firms and professional services firms are already using various approaches here. They’re not as old-school as they’re made out to be. But if clients know that the work can be produced faster, and have access to similar models, they may naturally expect part of the efficiency gain to be reflected in the price.

So, while AI increases the volume and speed of production below the senior lawyer, clients are applying pressure to time and price from above.

Clients are adding pressure as well

I call this the senior squeeze.

From below comes more AI-assisted production: more drafts, more issues and more work requiring verification.

From above comes client pressure: faster delivery, lower prices and an expectation that AI efficiencies will be passed on.

In the middle sits the senior lawyer.

Their attention has not become scalable at the same pace as LLMs generate text. Their professional responsibility has not been reduced (yet) and it is still their name, or the firm’s name, that goes on the final work.

Senior lawyers are therefore being asked to review more output, more quickly and at a lower price, while remaining accountable for the result. Ouch!

How do we solve this

To be continued next week…