I struggle to believe at an intuitive level that autonomous R&D is so ingenious. I have been monitoring a few side projects:

1. Building a new SOTA engine for Scrabble (current SOTA is not really a neural-network method and doesn't beat the strongest human player)
2. Building a bot to play Ultimate Tic-Tac-Toe
3. Researching philosophy and formalizing some proofs for the AI Philosophy contest

In all of these cases, the model's judgment is just very poor in a basic way. For example:

1. When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
2. It will conduct experiments that have no information value because the amount of training data used was obviously too small to distinguish the architectures to be compared.
3. In the philosophy project, where it built a wiki over many tokens and investigations, it kept going down directions which were extremely trivial and had little value.
4. For Ultimate Tic-Tac-Toe, I saw a very clever idea in a blog post that I doubt Astra would ever have come up with; it actually located the blog post but didn't realize its importance or relevance (which could be discerned easily from his strong performance in an online arena and the recency of the post) over papers from previous years that are obviously less relevant.

I judged an Apart hackathon last weekend and reviewed eight papers, all of which scored between 60% and 95% on Pangram. Predictably, the papers were often very similar to each other, and kind of methodologically incomplete in a vague but similar way, where they seem to overlook the most straightforward analysis.

Maybe the important exception is certain hill-climbing tasks with fast feedback loops. For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every method in the literature that seems kind of related," etc. But even in that case, two big leaps were when I said that it should take into account the fact that you can measure across a game rather than per random position, and suggesting that it try to design algorithms for three different cases based on the number of blank tiles left in the bag (which cause the number of moves to explode).

There is also the fact that Astra at least (and presumably Claude) tends to write reports that are extremely difficult to understand and some kind of Q&A interrogation is often needed to figure out what it actually did and what the key results are. Maybe there's a temptation to just say "whatever" and let it rip.