The Price of Innovation

Of course AIs will create their own language

What do we want our technology to optimize for?

Should we optimize for possibility? Or should we optimize for certainty?

The tradeoffs are escalating.

One fascinating aspect from the DeepSeek-R1 paper was how its emergent reasoning capacities began to mix languages. This may seem like a mere curiosity. Perhaps just a temporary side effect of AI innovation. But I think it points to a deeper dialectic that we need to confront.

One AI’s bug is another AI’s feature

While training models to develop reasoning capabilities, DeepSeek researchers discovered that the AI began to mix languages in unexpected ways. The models achieved strong results on reasoning benchmarks, but a tendency to switch between languages made their outputs less user-friendly.

The researchers found a way to maintain language consistency, but it came with a tradeoff. They added a specific reward system to use the same language, but the result was that the models performed slightly worse on reasoning. They optimized for certainty at the expense of possibility.

But why even make that trade-off? Why consider language mixing a bug at all?

Mixing different languages together is a bug if you are optimizing for certainty. No one is going to use a chatbot where it starts using a language they do not understand. No one is going to put their career on the line based on a reasoning output that used a mishmash of languages to arrive at its conclusion. Economic certainty will require some minimum maintenance of language consistency.

But mixing languages is a feature if you are optimizing for possibility. If you want to maximize reasoning capacity, why constrain yourself to one language? Why not use that weird German word when it can captures the information contained in four English sentences?

What you consider a bug or a feature depends entirely on what you are optimizing for.

A language of pure possibility

Let’s take the optimization of possibility to its logical conclusion. There is zero reason to think that any optimally reasoning agent would ever limit its capacity to reason to a single language. In fact, there is no reason to think such an agent would limit its capacity to a language legible to humans at all.

If you are an AI tasked with optimizing your reasoning capacity, the first thing to optimize is the language you are reasoning with. As you discover how different structural aspects of language impact different reasoning parameters, you’ll soon run into the hard limits of that language. It won’t take long to reach the point where the only path left to leverage language for greater reasoning will be to design your own.

Such a language would need to contain arguments of vast complexity, range over an entire corpus of knowledge, and consider an almost infinite set of hypothetical projections. It’s easy to imagine how this new language could unlock tremendous feats of reasoning.

It’s even easier to imagine how such a language would be completely incomprehensible to human judgement.

Pick your future

So what do we optimize for? Certainty or possibility?

Optimizing for certainty will mean the reliable production of consistent, predictable, and immediately useful outputs that conform to human expectations. Optimizing for possibility will mean the potential for novel, unprecedented capabilities and insights, even if they initially seem chaotic or illegible to human users.

We can apply this to the case of AI reasoning. If you demand that the norms of human language always be maintained in any AI agent, you are either shutting down all possibility or limiting its realization. In a very real way, you are limiting what life can explore and what humans can be.

Yet if you demand that AI reasoning develop with zero constraints, you are risking it becoming completely illegible to human judgement. At some point this would mean the loss of any human agency to judge its output, or even to understand it.

You could thread the needle and claim that advanced reasoning would by definition include the ability to optimize for its own certainty. If reasoning helps make possibility more legible to human judgement, then surely we must keep developing advanced reasoning! Yet if the history of technology has taught us anything, it’s that the law of unintended consequence is not one we want to bet against.

You could also reject the premise. Some see pure possibility as pure liberation. AI reasoning will be so superior that making any real decisions will no longer be an obligation. Finally, all human judgement will be purely optional. We will be mercifully absolved from all responsibility. And who really cares if AI reasoning is legible to human judgement if it’s solving disease and unlocking economic growth? If this is your desired future, then good luck merging with the machine.

For those of us holding out hope for a better future—one that can combine exponential tech with human judgement—treating this as an either/or decision leads to outcomes that no one wants. We need to navigate both ends of this optimization spectrum.

Embrace the dialectic

There is an alternative, but it is a painful one. Any path to a viable technological future will relentlessly commit to the dialectic between certainty and possibility.

In other words, any pursuit to optimize for possibility must combine an equal if not greater commitment to make that possibility legible to human judgement.

Yes, this means that every technological breakthrough—especially those that will radically expand possibility—may require equally radical breakthroughs. We may need advanced technologies to ensure that we can deploy other advanced technologies according to human judgement. Innovations will be needed to ensure that our judgement can capture the increasing dimensionality of our future.

This doesn’t mean conforming to static definitions or requiring absolute certainty. No such thing will ever exist. Our notions of certainty and possibility will need to evolve with the technologies that are pushing to redefine them.

Yet the test for failing this dialectic will remain unchanged. If we find technology increasingly escaping our capacity to judge it, or to conform to our judgements, or to even be intelligible to the entire realm of judgement— then we have ceded far too much in the pursuit of possibility.

Navigating this dialectic will not be easy. But when it comes to exponential technology, it is the price of innovation.