🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

Here’s today at a glance:

• Intelligence Explosion: Part B

• AI artwork of the day

Intelligence Explosion: Part B

Preamble

The thesis of Leopold Aschenbrenner, 2x OpenAI researcher fired in April 2024, in his document Situational Awareness was briefly:

• a. The Intelligence Explosion Is Going To Happen

• b. It Will Create Dangers

• c. Therefore, We Must Control It In This Prescribed Way

I covered Part A on Monday, in which Leo uses the scaling hypothesis to predict that a 10,000x (4 Orders Of Magnitude, or OOMs) increase in “Effective Compute“ will lead to “Automated AI Researcher/Engineer“ by 2028 and thereafter an Intelligence Explosion into Superintelligence as these automated researchers figure everything else out. Final Verdict Part A: Consistent with Ray Kurzweil predictions, but would require many, many coincident and prerequisite innovations in a very short timeframe to make it work.

Now let’s consider Part B.

B) It Will Create Dangers

The second part of the argument goes like this:

• 1. We will see incredible economic and social impact from the intelligence ramp

• 2. Therefore, all countries will realize this is a critical technology for the military

• 3. So, China will try to steal all the AI secrets

You only come to the second part of this argument given that the intelligence ramp happens. There is still a great deal of uncertainty right now because of the capability overhang from GPT-5/Sora/Voice Engine. My take a month ago:

1. Will we see incredible economic and social impact from the intelligence ramp?

Here’s Leo:

I don’t like the “naive doubling“ formulation, but I could definitely see a successful Siri-integrated ChatGPT with voice hitting a $10 billion revenue run rate sometime in 2025.

It is really the next doubling to $100 billion that is difficult to imagine. For this to happen, there will be a combination of growth and displacement. One would need to see for eg:

• Google losing its dominant search market share on iPhones to ChatGPT within 18 months.

• ChatGPT becoming the number 1 app anyone uses on their phone, the iPhone of apps, the app that consumes all the functions of all the other apps

• ChatGPT voice companions displacing dating apps as the primary place to search for companionship

• ChatGPT becoming the dominant replacement for games.

• Agents carrying out tasks becoming the dominant users of the Internet (similar to how bots trading stocks are the dominant users of financial exchanges)

• Most current human digital work will be performed by these agents

All this in roughly 3 years! Yes, it can happen, but boy, does it mean crazy competition in the tech sector to avoid becoming roadkill.

2. Therefore, all countries will realize this is a critical technology for the military

By 2027/2028, the enormous progress should be clear to all, and countries will be competing to build the trillion-dollar, 100-GW, Superintelligence Cluster. This Cluster will automate AI research in 2028/2029. It is essentially the last innovation humans will need to make.

At this point, the AI will rapidly:

• solve robotics

• accelerate scientific progress in all fields

• cause an industrial and economic explosion

• provide overwhelming military advantage

Given the advances that this intelligence will promise, the US and China will enter a period of Great Power Competition. Which leads to

3. China will try to steal all the AI secrets

Leo suggests the Chinese are about 2 years behind the US. He doesn’t think they have a chance of catching up fair and square, and so imagines they will deploy thousands of spies, billions of dollars in order to get the AI secrets and know-how.

Commentary

Other factors, including price, will slow growth

This is where things get weird.

It is plausible for OpenAI to hit a $10 billion revenue run rate in 2025, especially by being the underlying for Siri. Google pays almost $30 billion to Apple to be the default search engine and has a ~40% margin, so presumably makes $50 billion on iPhone searches. Peeling off 50% of those searches via voice at the same margin would result in $10 billion net revenue.

But things then go into Minsky bubble mode, as they always do when a company/industry grows this quickly. Competitors are launched, acquihires are made, and $100 million seed funding offers for 2-year veterans from OpenAI. Capital flows into less likely ideas, prices get bid up. The natural tendency of the market is to pull talent and capital into ideas that show progress and push it beyond the natural growth rate.

This has implications for wages, Bay Area real estate prices, global commodities, and the like. As the industry becomes bigger, calls for regulation grow, “they must be cheating“ industries under threat cry. The seeds of this are already present in copyright lawsuits from artists and creators and proposed AI regulation laws in every state.

And not to forget, wherever the rubber meets the road, where physical construction of nuclear power plants, robotic parts, and semiconductor fabrication equipment is required, it takes time to license, certify, validate, and deploy new products.

In short, there are naturally occurring forces that will attempt to strangle or moderate growth.

This technology is not universally attractive to all countries

Why do highly skilled Indians and Chinese migrate to the United States? The undeniable fact is that there is surplus intelligent individuals in those countries, who may, in fact, create social unrest (high caste Indians agitating against affirmative action for universities in India, or educated Chinese second-guessing the CCP) if they were to remain in those countries.

So while countries desire economic growth, most of the world is happy to experience catch-up growth, with the United States taking the brunt of technology shocks in each wave, then carefully picking and choosing US innovation to introduce into their countries.

Xi Jinping’s China is famously anti-software and anti-gaming. While they are hurriedly trying to build an indigenous semiconductor industry, the anti-gaming stance has locked them out of the market that Nvidia grew up in. Verbal chatbots have faced much pushback and regulation already, and entrepreneurs are cautious as they cannot guarantee Tiananmen or other sensitive topic censorship. AI might just join gaming, Bitcoin mining, online lending, and online education in the persona non grata zone.

In short, I am doubtful there will be this race to super intelligence. All politics are local, and each country will absorb the innovation in a manner and pace that it wishes to. In China, this will specifically mean not allowing dilution of CCP power. Given that one generation of entrepreneurs like Jack Ma has already been brought to heel, I find it unlikely that the powers that be will allow rapid flourishing of another generation of technology that would displace those in power.

China will try to steal the AI secrets

Chinese open source is about 3 months behind US open source (Alibaba Qwen2 release in June 2024 vs Meta Llama3 in April 2024). Chinese closed source is non-existent for now. GPT-4 release date was March 2023. Assuming OpenAI has kept the lead, China is probably 15 months behind the US, so a 2-year assumption is not a bad one.

Will China try to steal the AI secrets? I mean, sure. If you’re giving them away, they’ll take them. International relations are a subtle back-and-forth of “If we do this, are they going to be pissed enough to go to war or do something stupid?”. China is supplying Russia with munitions and technology for the war in Ukraine. And the US is not doing much about it.

So sure, China will try to steal the secrets. But the how and the how much matters. If they copy the hard disk of someone visiting China, well, that’s just fair play on bad opsec. If they assassinate an American AI researcher in San Francisco the way the US assassinated Iranian nuclear researchers in Tehran.. well that’s an act of war.

So I think the important question is not whether China will try to steal the AI secrets, but really how bad the US-China rivalry over AI will get.

Finally, do the stolen weights even matter? The next version of the model exists in the minds of researchers, and next month’s model is always better than this month's. In fact, it’s at least 21% better (1 OOM according to Leo), assuming the Notably due to the exponential curve, it is multiple times better. To get ahead of the curve, it might be better to steal the researchers rather than steal the weights…

Final Verdict Part B: Given the ramp to superintelligence, the geopolitical implications are unclear. A China-US race is possible, but so is a clampdown in China on a technology that threatens CCP supremacy, as has happened for Bitcoin mining, gaming, fintech, and many other sectors.

Share this story

🖼️ AI Artwork Of The Day

That’s it for today! Become a subscriber for daily breakdowns of what’s happening in the AI world: