**ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing** **Authors:** Lu Sun, Yun Seo Chang, Aaron Chan, Steven P. Dow **Abstract:** Peer review is a cornerstone of scientific research, but the increasing number of submissions has created a growing demand for reviewers. This study explores how technology can support novice reviewers in adhering to community standards and practices. Through a formative study with 10 novices and 10 experts, we identified challenges faced by novices and strategies used by experts. We developed ReviewFlow, an AI-driven workflow that provides contextual reflections, in-situ knowledge support, and notes-to-outline synthesis to help novices produce more comprehensive and structured reviews. A within-subjects experiment with 16 inexperienced reviewers showed that ReviewFlow led to more detailed and constructive reviews, though they still struggled to provide actionable suggestions. Participants appreciated the streamlined process but expressed concerns about using AI in scientific review processes. **Key Contributions:** 1. **Formative Study:** Revealed challenges faced by novices and strategies used by experts. 2. **ReviewFlow Development:** Designed a platform that incorporates intelligent scaffolding to support novice reviewers. 3. **Empirical Insights:** Showed how intelligent scaffolding can help novices write well-structured and comprehensive reviews. **Design Goals:** 1. **Contextual Cues:** Provide lightweight and adaptive scaffolds for reflection. 2. **In-situ Citation Support:** Offer knowledge support to assess novelty. 3. **Model Expert Workflow:** Structure the review process based on expert practices. 4. **Align with Community Standards:** Guide reviewers to produce reviews that meet community expectations. **System Implementation:** - **Contextual Cues:** Generate questions based on community criteria and paper content. - **In-situ Citation Recommendation:** Provide summaries and recommendations for cited papers. - **Notes-to-Outline Synthesis:** Summarize notes into outlines and facilitate drafting. - **Fact-Checking:** Ensure reviews are justified and align with community standards. **Method:** - **Participants:** 16 novice reviewers with varying levels of experience. - **Procedure:** Conducted a within-subjects experiment comparing ReviewFlow and a baseline interface. - **Results:** Showed that ReviewFlow improved review quality and engagement, but participants still faced challenges in providing actionable suggestions. **Conclusion:** ReviewFlow effectively supports novice reviewers by providing intelligent scaffolding, but further research is needed to address the limitations and ensure the integrity of the review process.**ReviewFlow: Intelligent Scaffolding to Support Academic Peer Reviewing** **Authors:** Lu Sun, Yun Seo Chang, Aaron Chan, Steven P. Dow **Abstract:** Peer review is a cornerstone of scientific research, but the increasing number of submissions has created a growing demand for reviewers. This study explores how technology can support novice reviewers in adhering to community standards and practices. Through a formative study with 10 novices and 10 experts, we identified challenges faced by novices and strategies used by experts. We developed ReviewFlow, an AI-driven workflow that provides contextual reflections, in-situ knowledge support, and notes-to-outline synthesis to help novices produce more comprehensive and structured reviews. A within-subjects experiment with 16 inexperienced reviewers showed that ReviewFlow led to more detailed and constructive reviews, though they still struggled to provide actionable suggestions. Participants appreciated the streamlined process but expressed concerns about using AI in scientific review processes. **Key Contributions:** 1. **Formative Study:** Revealed challenges faced by novices and strategies used by experts. 2. **ReviewFlow Development:** Designed a platform that incorporates intelligent scaffolding to support novice reviewers. 3. **Empirical Insights:** Showed how intelligent scaffolding can help novices write well-structured and comprehensive reviews. **Design Goals:** 1. **Contextual Cues:** Provide lightweight and adaptive scaffolds for reflection. 2. **In-situ Citation Support:** Offer knowledge support to assess novelty. 3. **Model Expert Workflow:** Structure the review process based on expert practices. 4. **Align with Community Standards:** Guide reviewers to produce reviews that meet community expectations. **System Implementation:** - **Contextual Cues:** Generate questions based on community criteria and paper content. - **In-situ Citation Recommendation:** Provide summaries and recommendations for cited papers. - **Notes-to-Outline Synthesis:** Summarize notes into outlines and facilitate drafting. - **Fact-Checking:** Ensure reviews are justified and align with community standards. **Method:** - **Participants:** 16 novice reviewers with varying levels of experience. - **Procedure:** Conducted a within-subjects experiment comparing ReviewFlow and a baseline interface. - **Results:** Showed that ReviewFlow improved review quality and engagement, but participants still faced challenges in providing actionable suggestions. **Conclusion:** ReviewFlow effectively supports novice reviewers by providing intelligent scaffolding, but further research is needed to address the limitations and ensure the integrity of the review process.