Unbundling Google

🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

When you book a trip:

• 25% of your hotel room price gets paid to an OTA

• Online Travel Agents like Expedia take that revenue and use it to advertise on Google

• 20% of Google’s revenue comes from the travel sector

This relationship is so significant that small changes on the Search Engine Results Page have a huge impact, as evinced by Expedia owner Barry Diller’s rant at such a change.

AI that plans a trip is one of the holy grails of product. And now the team at Ohio State University has finally created a benchmark to specifically measure how close we are to creating a Google killer, an AI travel planner.

The benchmark consists of a dataset of training set of 5 queries, with human-annotated solutions, and a test set of 1,000 queries.

The Agent has presumed access to 6 search tools:

• City

• Flights

• Distance

• Attraction

• Accommodation

• Restaurant

And the dataset includes 4 million entries for possible results from the 6 tools. In effect, this is the first closed-world dataset for travel planning which should be large enough to create a planner.

The findings using this dataset and the benchmark were.. interesting:

• LLMs are terrible at planning right now - GPT-4 had a success rate of 0.6%, while all other models totally failed

• Existing planning strategies fail at this level of complexity.

Google certainly doesn’t seem to need to worry yet… However, in AI, the creation of a good benchmark and dataset is often the first step towards finding the path to a solution… And I‘m sure we’ll be talking about state-of-the-art performance on this benchmark within a year.

Become a subscriber for daily breakdowns of what’s happening in the AI world: