Tweets over Peer Review

🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

📱 Tweets over Peer Review

You’ve done everything right. Waited 18 months for peer review, answered nonsense comments, submitted to a prestigious conference…Then you find out, submitting your paper to a preprint server, and getting tweeted by an “academic influencer“ would have been a better use of your time. Or at least that’s what this paper says:

Measuring citation count, it finds:

• Mention by Twitter users @_akhaliq (who does this professionally for Hugging Face as of last year) and @arankomatsuzaki (working on EleutherAI) had a 2x-3x impact on citation count

• They serve as curators for the entire ML/AI research space

I often feel like current-day academia engages in cargo cult rituals around fundamental information creation, quality control, curation, and dissemination. It’s good to start to see these rituals decay. The paper’s authors recommend:

• Re-evaluation of traditional models of paper selection and review - OK

• Evolve - towards what?

The emergent system here seems to be to

• push out research quickly to preprint servers

• include datasets, and other materials so that reviewers can analyze data for themselves rather than just relying on your conclusions

• publicize amongst the research community on social media and other platforms

• receive and respond to feedback in the open in real time

The advantages this system has are in speed, lack of fashionable gatekeepers, and greater attention and scrutiny to important papers with meaningful results. The disadvantages are in clear Pareto winner-take-most dynamics and a Gartner hype cycle around research that may obscure how useful it really is.

Meanwhile, the MAMBA (potentially the next big architecture after the current one) authors get rejected from ICLR 2024:

Aa Yann LeCun had said before:

In general, the problems around curation, around knowing what is important and what to pay attention to, only become ever more pressing as the flood of information ascends the elbow of the exponential curve. We are finally at the stage where the existing gatekeepers have been completely overwhelmed, and new ways of allocating attention are required.

Become a subscriber for daily breakdowns of what’s happening in the AI world: