ICLR's LLM Feedback: A Double-Edged Sword
The International Conference on Learning Representations (ICLR) has introduced a new dimension to its review process with LLM-generated feedback. While intended to streamline and enhance the quality of reviews, the experience for authors appears to be a mixed bag. Many researchers report receiving feedback that, while ultimately helpful, often includes a significant amount of trivial or overly detailed nitpicking alongside substantive points. One author noted receiving only 1-2 valid points across three pages of feedback, a sentiment echoed by others grappling with similar experiences.
This deluge of detailed, sometimes granular, feedback presents a unique challenge. Authors with the time and inclination can address both the critical and the minor points, potentially leading to paper improvements. However, it raises questions about the scalability and efficiency of such a system, particularly for those with less bandwidth to meticulously comb through every suggested change. The initiative is generally viewed as positive and capable of improving papers, but the signal-to-noise ratio in the feedback remains a point of discussion.
Submission Deluge and Reviewer Burden
Compounding the feedback experience is the sheer volume of submissions ICLR is experiencing. Reports indicate submission numbers nearing 51,000, a staggering figure that inevitably strains the review process. This surge in submissions directly impacts the quality and thoroughness of reviews, potentially leading to the kind of mixed feedback researchers are observing. When reviewers are overwhelmed, distinguishing between essential critiques and minor stylistic suggestions becomes more difficult, both for the reviewer and subsequently for the author parsing the feedback.
The sheer scale of submissions, with one user reporting their number close to 51k just 13 hours before abstract submission closure, highlights the immense pressure on the conference's infrastructure and its reviewer pool. This creates a bottleneck where valuable insights might be buried under an avalanche of less critical comments. The question then becomes how to effectively manage such large-scale academic evaluations without sacrificing the depth and accuracy of peer review.
Concerns Over Review Policy and Qualifications
Beyond the feedback itself, ICLR's review policy has also drawn scrutiny. A specific concern has been raised regarding the policy that mandates authors to serve as reviewers if their name appears on three or more papers. The ambiguity lies in the lack of explicit qualification requirements. This raises a scenario where a junior student, perhaps a fourth author on multiple papers, could be tasked with reviewing complex research without possessing the necessary expertise. This could lead to superficial or unqualified reviews, undermining the integrity of the peer-review system.
While the policy does not apply to authors with fewer than three papers, it highlights a potential systemic issue. The concern is not about the obligation to review, but about ensuring that those performing reviews are adequately equipped to do so. Without clear qualification guidelines, the system risks placing inexperienced individuals in positions where they must evaluate highly specialized research. This could result in inaccurate assessments and a dilution of the review standards that are critical for maintaining academic rigor.
Transparency and Future Considerations
A lesser-discussed but significant aspect of the ICLR feedback process is transparency. Authors are seeking clarity on whether reviews will remain public. While this can foster accountability and learning, a lack of prior warning about public reviews can be jarring. Researchers appreciate constructive criticism, but receiving it in a format that might be publicly scrutinized without adequate preparation can be an uncomfortable experience. Clear communication about the public nature of reviews is essential for managing author expectations and fostering a more supportive review environment.
The initiative to use LLMs for feedback is innovative, aiming to improve paper quality. However, the current experience suggests a need for refinement. Balancing the volume of feedback, ensuring reviewer qualifications, and maintaining transparency are key challenges. As ICLR continues to evolve, addressing these points will be crucial for maximizing the benefits of its advanced review systems while mitigating potential drawbacks for the research community.