Learn about this first
Ai2 released the study “Old Ideas, Novel Problems,” testing whether large language models can reliably judge the novelty of research ideas. Ai2 said the results show that this type of novelty judging is not stable.
Also released alongside the paper are a benchmark, data, and code for evaluating novelty judging, as well as an automated pipeline for researchers to build new evaluation sets.
AI-generated from reports · updated 20 days agoLatest developments
Ai2 Releases Old Ideas, Novel Problems Research and Benchmark to Test the Reliability of LLMs in Judging the Novelty of Research Ideas
Event progress1
Ai2 Releases Research and Benchmark on the Reliability of LLM Novelty Evaluation
Ai2 / Allen Institute for AI: Ai2 Releases the Old Ideas, Novel Problems Study and Benchmark, Revealing That LLM Novelty Judging Is Unstable