- Emergent Behavior
- Posts
- Just Keep Asking
Just Keep Asking
Does increasing the number of AI agents improve their performance on a task?
đź”· Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/0e0158b6-b9cc-47dd-91d1-e485f8582d9b/Screenshot_2024-01-22_at_2.13.39_PM.png?t=1705950837)
This research paper from the Tencent China team tries to find out whether just organizing the existing AI models we have into teams of agents that work together on a problem improves their performance, i.e., squeezing greater intelligence from a crowd of similar intelligences.
Paper Title: More Agents Is All You Need
Who: Team from Tencent China
What did they study:
Does increasing the number of AI agents improve their performance on a task?
In essence, does brute force scaling of AI agents work?
How did they do it?
Create agents that in perform language model requests in two phases
In query phase, the input is iteratively fed into the same LLM or multiple LLMs to get multiple outputs
In the voting phase, a majority vote is taken on the best response
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/049fea04-a34e-4ee1-bdcd-794d2bcdbb53/Screenshot_2024-02-10_at_1.47.52_PM.png?t=1707601688)
What did they find?
Accuracy increases with the number of agents, notably at an ensemble size of 15 (either making 15 iterative calls to the same model or having multiple agents work on the same query), a Llama2 can match GPT3.5 single query, and a GPT3.5 can match a GPT4 single query.
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/66e9e9e9-b8d5-4caa-89b7-e9346e64f144/Screenshot_2024-02-10_at_1.49.24_PM.png?t=1707601803)
Adding agents to the ensemble increased accuracy in arithmetic reasoning
Performance gains increase initially with increasing difficulty but then hit a ceiling and then decline as problem complexity overwhelms the model
Gains increase with the number of steps or iterations
What are the implications?
You can increase intelligence with the same AI model just by asking the same question over and over iteratively
This technique can improve raw performance for all AI/LLMs
Brute force scaling works to a point but hits a ceiling
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/0e0158b6-b9cc-47dd-91d1-e485f8582d9b/Screenshot_2024-01-22_at_2.13.39_PM.png?t=1705950837)
Become a subscriber for daily breakdowns of what’s happening in the AI world:
Reply