ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing

March 18-21, 2024 | Lu Sun, Aaron Chan, Yun Seo Chang, Steven P. Dow
ReviewFlow is an AI-driven system designed to support novice reviewers in academic peer reviewing by providing intelligent scaffolding. The study aimed to understand the challenges faced by novices and the strategies used by experts in peer reviewing. A formative study with 10 novices and 10 experts revealed that novices lack guidance on how to structure and assess peer reviews, while experts use a workflow involving critical reading, annotation, note-taking, and synthesis. ReviewFlow was developed to scaffold novices with contextual reflections, in-situ knowledge support, and notes-to-outline synthesis to align peer reviews with community standards. In a within-subjects experiment with 16 novice reviewers, participants using ReviewFlow produced more comprehensive reviews, identifying more pros and cons, but still struggled to provide actionable suggestions. Participants appreciated the streamlined process but expressed concerns about using AI in scientific review. The study highlights the potential of intelligent scaffolding to help novices write well-structured and comprehensive reviews while addressing the challenges of peer reviewing. ReviewFlow incorporates features such as contextual cues, in-situ citation recommendations, and notes-to-outline synthesis to support the peer review process. The system was implemented as a React web application with a Flask backend, using GROBID for PDF data extraction and GPT-4 for generating contextual cues. The study found that ReviewFlow improved review quality and engagement, but also highlighted the need for careful consideration of AI's role in peer reviewing to avoid bias and ensure accurate evaluations.ReviewFlow is an AI-driven system designed to support novice reviewers in academic peer reviewing by providing intelligent scaffolding. The study aimed to understand the challenges faced by novices and the strategies used by experts in peer reviewing. A formative study with 10 novices and 10 experts revealed that novices lack guidance on how to structure and assess peer reviews, while experts use a workflow involving critical reading, annotation, note-taking, and synthesis. ReviewFlow was developed to scaffold novices with contextual reflections, in-situ knowledge support, and notes-to-outline synthesis to align peer reviews with community standards. In a within-subjects experiment with 16 novice reviewers, participants using ReviewFlow produced more comprehensive reviews, identifying more pros and cons, but still struggled to provide actionable suggestions. Participants appreciated the streamlined process but expressed concerns about using AI in scientific review. The study highlights the potential of intelligent scaffolding to help novices write well-structured and comprehensive reviews while addressing the challenges of peer reviewing. ReviewFlow incorporates features such as contextual cues, in-situ citation recommendations, and notes-to-outline synthesis to support the peer review process. The system was implemented as a React web application with a Flask backend, using GROBID for PDF data extraction and GPT-4 for generating contextual cues. The study found that ReviewFlow improved review quality and engagement, but also highlighted the need for careful consideration of AI's role in peer reviewing to avoid bias and ensure accurate evaluations.
Reach us at info@study.space