2021-04-01: EB-3: Building Intelligent Game Worlds

🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

Here’s today at a glance:

• EB-3 - Building Intelligent Game Worlds

• Things happen

• AI artwork of the day

🎙️ EB-3 - Building Intelligent Game Worlds

Follow

EB-3, our third podcast, dropped today. Before I continue, the rules of the game are:

• Pods that CHART stays alive

• Pods that get a Follow on Apple Podcasts CHART

• Pods that get Listened to on any app CHART

So FIRST, CLICK on the link below (opens up your Apple Podcasts app), and click “+Follow” (in the upper right-hand corner)

Then go ahead and listen to the podcast any way you want to on your preferred app.

Listen on

• Apple Podcasts

• Spotify

• Youtube

• RSS

Who

Vishnu Hari, CEO (LinkedIn), and Peggy Wang, CTO (LinkedIn), founders of Ego, an AI native game engine that is part of the Y Combinator Winter 2024 batch. Peggy + Vish are ex-Meta AI and Reality Labs engineers who are huge game nerds.

What

Ego is an AI native simulation engine and platform where users can create and share 3D animated characters, worlds, and scripted interactions all using natural language. Imagine being able to prompt the Matrix, Sims, or GTA into existence. They built 2 simulations (TownWorld and CharacterIdol) with their engine during YC and are backed by notable angels and funds such as Immad Akhund (Mercury), Kayvon Beykpour (Periscope), BoostVC, and PearVC.

When

Feb 22, 2024

Background

There are three big problems in open-world games:

1. It is hard for humans to write code and script interactions between assets, characters, and the environment to design fun games and experiences

2. It is impossible for NPCs to display human-like behavior and have true agency in these 3D worlds

3. It is hard to create new 3D assets, worlds, and non-playable characters

Ego has built an AI game simulation engine to solve the first two of these problems.

Avatars

But before that, Ego initially launched an iOS app called Ego that allowed users to become 3D-generated avatars with high-quality face tracking. Key points about the Ego Live Avatar product:

1. Users could scan their faces and use a character creator to customize their 3D avatars.

2. The app utilized face tracking technology without requiring a depth sensor, making it accessible on more devices.

3. Peggy, who previously worked on face tracking at Meta Reality Labs, brought expertise to improve the avatar's expressiveness and face tracking quality.

4. The app featured a universal rig, allowing users to switch between different avatar characters on the fly during livestreams.

5. Ego aimed to enable pseudonymous 3D social interactions through these expressive avatars.

After launching the Ego app, they realized people cared more about avatar customization than face tracking quality and building customizations for avatars was not scalable at the time. As they were trying to figure out their next steps, they had a revelatory dinner with Emmett Shear, co-founder of Twitch (and former CEO of OpenAI).

Emmett advised them that what they were building downstream of 3D social spaces was actually a game, and they needed to build a game rather than just a chat app to succeed.

TownWorld

Around the same time, Joon Sung Park’s Stanford Generative Agents (aka Smallville), which simulated a small town using AI agents that could talk to each other such that they self-organized birthday parties and meetups, was released. Vish and Peggy were inspired by this to implement AI agents in 3D.

This became TownWorld

TownWorld features autonomous AI agents populating a hand-crafted 3D map

• AI generates all routines, conversations, and voices in real-time, while the team focuses on optimizing performance through multithreading and prompt optimization

• A seven-day simulation with three agents costs only $7 in OpenAI credits, thanks to efficient resource management

The team implemented a "world marker" system to enable seamless navigation and object interaction within the virtual town.

• Agents can navigate the environment, interact with objects, and keep track of their state and position

• AI-generated voices powered by Eleven Labs enhance the immersive experience

• Despite some quirks in the first version, the team has ambitious plans for future improvements, including added memory, more characters, and enhanced animations

Ego envisions an AI-native 3D simulation engine that empowers creators to generate entire virtual worlds using only natural language prompts.

• AI will generate all components, from 3D maps and environment assets to character animations and interaction scripts

• Creators can build games like TownWorld, The Sims, or Animal Crossing without any coding or animation skills

CharacterIdol

2 player dialogue game, where you can choose characters to converse with, built on the Ego framework.

Next up, the team presents at YC! Good luck, guys!

Transcript

Part 1

Part 2

Part 3

Links

Share this story

🗞️ Things Happen

• ChatGPT (the 3.5 free version) has been used by roughly a quarter of US adults, with usage intensity higher the more educated they are. God only knows what happens when GPT-4 goes free sometime this year.

🖼️ AI Artwork Of The Day

That’s it for today! Become a subscriber for daily breakdowns of what’s happening in the AI world: