The Dwarf, The Dragon, The Ludens and Marble

Tyrion Lannister immortalized the expression, “I drink and I know things.”

If I had to come up with a similarly laconic description of myself, it would be, “I read and I think about things.”

The core of this piece is inspired by books of two authors that every Soviet schoolboy knew by heart – the Strugatsky brothers.

They spent much of their careers thinking about a problem that is becoming increasingly relevant to us: what happens when human beings create, encounter, or become subject to forms of intelligence and systems that are more powerful than their ability to understand them?

Interestingly enough, as I found myself reflecting on their books, I could not resist the temptation to think about two more – The Dragon by Evgeny Schwartz and Marble by Joseph Brodsky.

An important caveat: I realize, of course, that the books I am writing about here are not part of the Western canon, and 95% of my network will not have read them. The closest parallels in the West are Lem and Asimov. I will do my best to explain why I think the books are relevant to what we all are thinking about.

Ready?

The Dragon: what if people choose their own chains?

The Dragon is a play about tyranny, and the truly disturbing (or even depressing) takeaway is that the people adapt to oppression. They learn to live with it, rationalize it, and eventually defend it. The darker interpretation is that submissiveness to authority is an integral part of human nature.

When the Dragon is defeated, the problem does not disappear. The inhabitants of the world described by Schwartz (which, frankly, is beginning to look suspiciously similar to the one we live in today) are perfectly happy for someone else to make the difficult choices, even if their freedom is unfettered.

That feels relevant to AI.

The obvious question is whether AI will take away human agency (something that Quentin Solt and I have been debating relentlessly in our #TrojanHorseDebate).

The more interesting question is whether we will voluntarily give it away.

In law, the temptation is obvious: let the machine find the cases, construct the argument, draft the document, identify the risk, recommend the strategy. The decision-making stays with the human (for now), but that boundary feels amorphous enough – and abdication of responsibility no longer feels like such an impossible leap of faith.

At some point, without being forced to surrender judgment, we may simply choose to stop exercising it.

The Predatory Things of the Age: what if technology gives us exactly what we want?

The Strugatskys’ The Predatory Things of the Age is even more unsettling.

In that book, technology is more of a siren song. It is not coercive – it is seductive.

It gives people entertainment, stimulation and gratification with extraordinary efficiency.

And when that mirror is placed in front of us, the uncomfortable realization is that people are very good at wanting things that are bad for them.

This is the question that Quentin has been posing (with his “Elon Musk as the Serpent tempting us all in the Garden of Eden” jab) – what is the price we will pay for abundance of intelligence and the surfeit of efficiency?

Predatory Things cautions that, with Musk’s apple in hand, we may prefer not to think for ourselves.

The Ugly Swans: what if the future really is better than us?

Then comes the harder problem.

The Ugly Swans shows us the world where – alongside ordinary humans – a new type of beings appears. They could care less about honors, riches or power – they are driven by creativity and desire for knowledge.

The book suggests that the temptation is irresistible, and the new generation will choose that path.

If AI becomes better than lawyers at legal reasoning, what is the argument for keeping humans in the loop?

It feels more like a preference or a value judgment. It may not be irrational, but we should be honest about what we are doing: we are choosing to preserve a human role even when the machine may perform it better.

One Billion Years Before the End of the World: what happens if we do not understand the system?

In One Billion Years Before the End of the World, scientists encounter something that seems to prevent them from advancing human knowledge, and the identity of that power remains a mystery. It just exists, like a law of nature.

The protagonists of that book do not know the rules of the system in which they find themselves.

That is a useful metaphor for AI.

A lawyer can receive an extraordinary answer from a model without knowing exactly why the model arrived there, what assumptions shaped it, what possibilities it discarded, or what it never considered.

That does not make the answer wrong, but it raises a fundamental question for legal AI:

Knowing an answer is not the same thing as knowing why you are entitled to trust it.

Justification, reasoning and accountability are (for now; cue Quentin for a comment) the keystone of the legal system.

The Doomed City: what if we are inside the experiment?

The Doomed City takes the problem further.

Its inhabitants live inside a system whose ultimate purpose they cannot understand. They build governments and overthrow them, the create ideologies and replace them, they convince themselves that they have the system figured out.

But – on the last page of the novel – the reader realizes that the inhabitants (and us) are still clueless about what the rules are.

What a metaphor for AI governance, eh?

Imagine a collection of individually reasonable systems whose aggregate behavior nobody fully understands.

AI is increasingly and rapidly becoming infrastructure, the connective tissue between lawyers and legal knowledge, between the decision-makers and the options they are choosing from.

And the system remains an enigma.

The Waves Extinguish the Wind: what if something genuinely new emerges?

And then the Strugatskys, expanding on the themes from The Ugly Swans, go one step further.

In The Waves Extinguish the Wind, humankind discovers that some humans are not human.

Biologically, socially, mentally, they are simply beyond humanity. Humanity’s concerns are as alien to them as the problems of the ants are to us.

We worry about machines attacking or supplanting humans, but there is a perfectly plausible scenario where intelligence no longer maps neatly onto ours (I suspect that this would require a different AI architecture, but someone like Devansh Devansh or Umar Butler would know better).

But here is where I depart from the more deterministic versions of this argument (and, in the process, arrive to the semi-strong version of my agency theory).

Seeing what is happening matters.

Awareness creates choice – what is automated and what is preserved; if not the system as a whole, then at least some elements of it. The future is happening to us, but – in a Soros-reflexivity kind of way – is still influenced by each of us.

And that brings me to Joseph Brodsky’s Marble

In Marble (probably the greatest play that remarkably few have heard about), history, politics, the grand struggle are a distant backdrop to two people sharing a (futuristic) prison cell. And the entire play is (essentially) a dialogue – about history, culture, relationships and the essence of humanity.

I think this could be the most important question that I am wrestling with in this piece: in the post-AI world, what remains after everything that can be optimized has been optimized?

AI is (or will soon be) better than us at every single task that has historically constituted the core of our professional competence – drafting, research, analysis, calculation.

But then there are things whose value is not aligned to any efficiency measure. (Shoutout to Angelina Canelli and Tatiana Botskina for brilliant reminders that one cannot place a price tag on a mission, on the sense of belonging, and other intangibles).

Relationships, individual memories, conversations, art – personal and interpersonal stuff – ultimately, these are the things that matter once everything else is stripped away.

···

Schwartz cautions that tyranny and submissiveness are ingrained in human nature (akin to the Original Sin, Quentin).

The Strugatsky brothers warn us of the perils of technology domination, seduction, incomprehensible systems and evolutionary discontinuity.

Brodsky (read through the lens of legal tech), in Marble, asks the quieter question:

When the machines can do almost everything that can be done (and can do that better than humans), what will we do because we still want to do it ourselves?

Perhaps that is the one question that subsumes all the others.