🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

Here’s today at a glance:

• Intelligence Explosion: Part A

• AI artwork of the day

Intelligence Explosion: Part A

Leopold Aschenbrenner, 2x OpenAI researcher fired in April 2024, was on the Dwarkesh podcast this week. Along with a 4-hour pod, he dropped a 160-page treatise on the coming Intelligence Explosion in the decade ahead, to expound on his Situational Awareness.

The backstory is that Leo was on the four-person team running the FTX Future Fund, backed by Sam Bankman-Fried. Dwarkesh receives a small $20k grant from the Future Fund right out of college, which encourages him to take up podcasting full-time. After FTX collapses, Leo joins the OpenAI superalignment effort. Dwarkesh and Leo become friends, and much of Dwarkesh’s “alpha“ on what to ask AI guests comes from this friendship.

The rest of Leo’s story is best summarized:

Situational Awareness

The Intelligence Explosion that Leo talks about is basically Kurzweil’s approach to AGI in 2029 and running up to the Singularity in 2045.

It’s crazy, but I have to applaud him for actually trying to tackle the problem more concretely than Kurzweil, who uses solid data from the past but handwaves to the future. The approach is important because it determines who makes money in the next two decades, and whose values end up getting propagated into the new era. Notably, the value of being first to AGI is what spurred Sam Altman to go full bore on OpenAI.

The argument is basically in three parts

• a. The Intelligence Explosion Is Going To Happen

• b. It Will Create Dangers

• c. Therefore, We Must Control It In This Prescribed Way

I would note that Leo’s arguments are hand-wavy because he’s from an academic economics background. He alludes to this in the podcast by calling himself a former adherent in the Efficient Market Hypothesis. This is pretty on the dot, as his AI advice is also in the same vein as that of the theorist over the practitioner. A financial markets trader or engineer would not be making these arguments, as your illusions meet reality on a daily basis.

In any case, let’s evaluate them.

A. The Intelligence Explosion Is Going To Happen

Let’s be honest, this was an artful post, truth mixed with artfully applied naivete, allowing people to dunk on him, but ramping up distribution (it’s now at 7.5 million views, with a Twitter DAU of 28 million). You may assume a quarter of the extremely online world has seen this by now. Anyway

Unclear definition of intelligence

First, Leo does not define “intelligence“. He instead uses vague labels:

• “preschooler” (GPT-2)

• “Elementary schooler” (GPT-3)

• “smart high-schooler” (GPT-4)

• “automated AI researcher/engineer” (AI in 2028)

The intelligence explosion is supposedly when the “automated AI researcher/engineer“ level is reached in 2028/2029, beyond which we could see a rapid increase to “superintelligence“.

Orders of Magnitude (OOMs)

He then “counts the OOMs“ (Orders Of Magnitude) of increase in scale of “Effective Compute“ from:

• increased raw compute (~0.5 OOM per year) => TFLOPS at training time

• algorithmic efficiencies (~0.5 OOM per year) => Methods used in training such as the transformer architecture and inference

• “unhobbling” (not estimated) => Chain-of-Thought prompting and other techniques

Commentary

Deep breath: Imprecise but not totally wrong.

The Scaling Hypothesis

Let’s review OpenAI’s original Scaling Hypothesis:

In the original scaling hypothesis, the outcome of an increase in scale of data, model size, and compute was a decrease in test loss.

In the GPT-4 paper, test loss on the codebase next word prediction task became a proxy for intelligence:

That this viewpoint was acceptable by folks in the company was confirmed:

Factors leading to this increase in capability

1. Increase in raw compute

Jensen Huang’s target at Nvidia is a 1 million-fold (or 6 OOM) improvement over the next 10 years to match the progress of the last 10 years. This works out to 0.6 OOM per year, or slightly over Leo’s 0.5. So while this may be an erroneous assumption, it is the market assumption at this point.

TSMC forecasts 3 OOMs in 15 years, or 0.2 OOMs per year on GPU performance per watt. Put another way, with no additional buildout of power, we can expect 0.2 OOMs improvement.

Put some increase in power per TFLOP on the table, and yes 0.5 OOMs per year sounds plausible.

2. Algorithmic Efficiencies

Anyone with exposure to AI right now can probably point to dozens of minor algorithmic efficiencies that have been discovered in the past year. I can’t say for sure that it’s predictable that 0.5 OOM per year of advances are available, but it doesn’t seem outlandish. There are dozens of ideas like the below being regularly attempted, some succeed and progress continues.

There’s also stuff like the below floating around, which would invalidate Nvidia chips in favor of Field Programmable Gate Arrays from firms like Xilinx.

3. Unhobbling

Sure. A non-measurable, non-impactful factor on the graph. Irrelevant, but sure. We’ve seen that there is still lots of room for optimization of various kinds, even at the data center, in this post from former Google Brain research scientist and current Reka Labs founder Yi Tay.

Effective At What?

Where the argument falls apart is whether this creates “intelligence”, an “automated AI researcher”, which then recursively self-improves to infinity and beyond.

Unlike the scaling hypothesis scientific test loss metric, Leo does not have a clear idea of what this OOM increase in effective compute is supposed to produce. Without a clear goal, in fact, you cannot even tell whether the compute was in fact effective, as in, did the spending on the compute move you towards the desired outcome?

The blithe assumption that GPT-4 is a “smart high schooler” level of intelligence centers around the model’s ability to pass the SATs and AP Exams. However, the model is also unable to complete a hotel or flight booking. It’s generation capabilities put it in the calculator category: an interesting tool, but not intelligent.

This flaw—the lack of definition of intelligence feeding back into the nature of how “effective” the compute actually is—dooms the analysis from the get-go.

For argument’s sake, let’s assume test loss is the metric as per the scaling hypothesis. Let’s assume that through several OOMs the test loss continues to get smaller and smaller. You get perfect scores on the MMLU.

Can the model book a flight ticket now? Would you hand it your credit card?

Would you call this “AGI“ or “superintelligence” if it could not?

In Leo’s view, the automated AI researcher overcomes the talent bottleneck that AI research currently faces. And once AI research is automated, it automates research for everything else.

Bravo! But how does the AI researcher find data to invent quantum computers for the next order of magnitude improvement, for example?

Through experimentation and hypothesis testing. In the real world, this is slow, even with robots.

And as you scale, you may need many, many other innovations: room-temperature superconductors, molecular assembly, femtotechnology, etc… the list continues.

This is where the exponential turns into a sigmoid unless a whole lot of other things happen in the real world.

The assumptions inherent in that intelligence explosion step are numerous. This is about where Kurzweil is too. Leo is pretty consistently within the Kurzweil window, and though we may all smirk at his predictions, they have been eerily correct, as even Geoff Hinton has acknowledged.

Final Verdict Part A: Consistent with Kurzweil, but would require many, many coincident and prerequisite innovations in a very short timeframe to make it work.

Postscript: This newsletter has started to hit the limits of Gmail, so I will cover Parts B and C in subsequent days. Stay tuned!

Share this story

🖼️ AI Artwork Of The Day

That’s it for today! Become a subscriber for daily breakdowns of what’s happening in the AI world: