Source time Edition date
Close readHugging Face
Can we trust LLMs to judge whether a research idea is novel?
我们能信任LLM判断研究想法是否新颖吗？
Worth reading because trusting LLMs to judge research novelty raises a real tradeoff: cheaper, faster reviews may come at the cost of reliability that researchers must decide whether to accept.
Judgment
Research teams and venue reviewers deciding whether to let an LLM screen ideas for novelty must weigh faster, cheaper triage against Ai2's finding that rewording a prompt moves a judge's accuracy by more than 50 percentage points — the configuration they pick, not the model alone, sets the reliability they accept.
Basis
Ai2's write-up reports that across six LLM judges — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6 — small changes to the evaluation setup produced dramatically different verdicts on the same ideas; changing just a few words in a prompt shifted accuracy at identifying which of two ideas is more novel by more than 50 percentage points. Retrieval and additional reasoning did not consistently improve judgments, and two purpose-built novelty evaluation systems performed worse than a simple prompted Claude Opus 4.6 baseline. The benchmark is built from ICLR 2026 reviews: a high-novelty pool of 154 accepted ideas from the top-rated 10% of their research area with undisputed novelty praise, a lower-novelty pool of 145 rejected ideas from the bottom 10% with originality concerns, and 154 LLM-generated ideas paired against human ones as weakly labeled comparisons. Mechanism: because verdicts hinge on model, prompt, evaluation format and reasoning budget, a team's own configuration — not a public leaderboard number — determines what its novelty judge will actually approve or reject.
- High-novelty human ideas154
- Lower-novelty human ideas145
- LLM-generated comparison ideas154
- Expert-reviewed pairs30
Unclear The source does not say which specific prompt rewording produced the 50-plus-point swing, nor report per-judge accuracy figures for either evaluation setting.
Next Check the released benchmark, data and code for per-judge accuracy on Human-Only versus Human+Generated, and whether any judge's edge over the prompted Claude Opus 4.6 baseline survives prompt rewording.