Two Peers Are Better than One: Aggregating Peer Reviews for Computing Assignments is Surprisingly Accurate
| dc.contributor.author | Reily, Ken | |
| dc.contributor.author | Finnerty, Pam Ludford | |
| dc.contributor.author | Terveen, Loren | |
| dc.date.accessioned | 2023-06-08T11:44:31Z | |
| dc.date.available | 2023-06-08T11:44:31Z | |
| dc.date.issued | 2009 | |
| dc.description.abstract | Scientific peer review, open source software development, wikis, and other domains use distributed review to improve quality of created content by providing feedback to the work's creator. Distributed review is used to assess or improve the quality of a work (e.g., an article). However, it can also provide learning benefits to the participants in the review process. We developed an online review system for beginning computer programming students; it gathers multiple anonymous peer reviews to give students feedback on their programming work. We deployed the system in an introductory programming class and evaluated it in a controlled study. We find that: peer reviews are accurate compared to an accepted evaluation standard, that students prefer reviews from other students with less experience than themselves, and that participating in a peer review process results in better learning outcomes. | en |
| dc.identifier.doi | 10.1145/1531674.1531692 | |
| dc.identifier.uri | https://dl.eusset.eu/handle/20.500.12015/4862 | |
| dc.language.iso | en | |
| dc.publisher | Association for Computing Machinery | |
| dc.relation.ispartof | Proceedings of the 2009 ACM International Conference on Supporting Group Work | |
| dc.subject | education | |
| dc.subject | collaboration | |
| dc.subject | peer review | |
| dc.title | Two Peers Are Better than One: Aggregating Peer Reviews for Computing Assignments is Surprisingly Accurate | en |
| gi.citation.publisherPlace | New York, NY, USA | |
| gi.citation.startPage | 115–124 | |
| gi.citations.count | 60 | |
| gi.citations.element | Tomás Alves, Francisco Sousa, Sandra Gama, Joaquim Jorge, Daniel Gonçalves (2023): How Personality Traits Affect Peer Assessment in Distance Learning, In: Technology, Knowledge and Learning 1(29), doi:10.1007/s10758-023-09694-2 | |
| gi.citations.element | Wilhelm A Friess, Andrew J Goupee (2019): Transformation of a Mechanical Engineering Capstone Experience, In: 2019 IEEE Frontiers in Education Conference (FIE), doi:10.1109/fie43999.2019.9028435 | |
| gi.citations.element | Milorad Tosic, Valentina Nejkovic (2010): Trust-Based Peer Assessment for Virtual Learning Systems, In: Lecture Notes in Computer Science, doi:10.1007/978-3-642-16567-2_13 | |
| gi.citations.element | Vivienne Farrell, Graham Farrell, Paul Kindler, Gilbert Ravalli, David Hall (2013): Capstone project online assessment tool without the paper work, In: Proceedings of the 18th ACM conference on Innovation and technology in computer science education, doi:10.1145/2462476.2462484 | |
| gi.citations.element | Yubo Kou, Colin M. Gray (2017): Supporting Distributed Critique through Interpretation and Sense-Making in an Online Creative Community, In: Proceedings of the ACM on Human-Computer Interaction CSCW(1), doi:10.1145/3134695 | |
| gi.citations.element | Anthony Estey, Jeremy Long, Bruce Gooch, Amy A. Gooch (2010): Investigating studio-based learning in a course on game design, In: Proceedings of the Fifth International Conference on the Foundations of Digital Games, doi:10.1145/1822348.1822357 | |
| gi.citations.element | Zuzana Kubincová, Martin Homola, Veronika Dropčová (2017): Peer Reviews in a Web Design Course: Now Students like Them Too, In: Lecture Notes in Computer Science, doi:10.1007/978-3-319-52836-6_66 | |
| gi.citations.element | Pantelis M. Papadopoulos, Thomas D. Lagkas, Stavros N. Demetriadis (2012): How to improve the peer review method: Free-selection vs assigned-pair protocol evaluated in a computer networking course, In: Computers & Education 2(59), doi:10.1016/j.compedu.2012.01.005 | |
| gi.citations.element | Joe Gibbs Politz, Shriram Krishnamurthi, Kathi Fisler (2014): In-flow peer-review of tests in test-first programming, In: Proceedings of the tenth annual conference on International computing education research, doi:10.1145/2632320.2632347 | |
| gi.citations.element | Christopher D. Hundhausen, Pawan Agarwal, Michael Trevisan (2011): Online vs. face-to-face pedagogical code reviews, In: Proceedings of the 42nd ACM technical symposium on Computer science education, doi:10.1145/1953163.1953201 | |
| gi.citations.element | Dilrukshi Gamage, Mark E. Whiting, Thejan Rajapakshe, Haritha Thilakarathne, Indika Perera, Shantha Fernando (2017): Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives, In: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale, doi:10.1145/3051457.3054013 | |
| gi.citations.element | Daniel Olivares, Christopher Hundhausen, Namrata Ray (2021): Designing IDE Interventions to Promote Social Interaction and Improved Programming Outcomes in Early Computing Courses, In: ACM Transactions on Computing Education 1(22), doi:10.1145/3453165 | |
| gi.citations.element | Ville Hyyrynen, Harri Hämäläinen, Jouni Ikonen, Jari Porras (2010): MyPeerReview, In: Proceedings of the 10th Koli Calling International Conference on Computing Education Research, doi:10.1145/1930464.1930481 | |
| gi.citations.element | Axel Abels, Tom Lenaerts, Vito Trianni, Ann Nowé (2020): How Expert Confidence Can Improve Collective Decision-Making in Contextual Multi-Armed Bandit Problems, In: Lecture Notes in Computer Science, doi:10.1007/978-3-030-63007-2_10 | |
| gi.citations.element | Dave Clarke, Tony Clear, Kathi Fisler, Matthias Hauswirth, Shriram Krishnamurthi, Joe Gibbs Politz, Ville Tirronen, Tobias Wrigstad (2014): In-Flow Peer Review, In: Proceedings of the Working Group Reports of the 2014 on Innovation & Technology in Computer Science Education Conference, doi:10.1145/2713609.2713612 | |
| gi.citations.element | Axel Abels, Tom Lenaerts, Vito Trianni, Ann Nowé (2020): Collective Decision-Making as a Contextual Multi-armed Bandit Problem, In: Lecture Notes in Computer Science, doi:10.1007/978-3-030-63007-2_9 | |
| gi.citations.element | Sandipan Sikdar, Matteo Marsili, Niloy Ganguly, Animesh Mukherjee (2017): Influence of Reviewer Interaction Network on Long-Term Citations: A Case Study of the Scientific Peer-Review System of the Journal of High Energy Physics, In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), doi:10.1109/jcdl.2017.7991572 | |
| gi.citations.element | Gail L. Ring (2015): Implementing a Peer Mentoring Model in the Clemson ePortfolio Program, In: Theory Into Practice 4(54), doi:10.1080/00405841.2015.1077616 | |
| gi.citations.element | Sharique Hasan, Rembrand Koning (2019): Conversations and idea generation: Evidence from a field experiment, In: Research Policy 9(48), doi:10.1016/j.respol.2019.103811 | |
| gi.citations.element | Juan Chen, Yingjun Cao, Linlin Du, Youwen Ouyang, Li Shen (2019): Improve Student Performance Using Moderated Two-Stage Projects, In: Proceedings of the ACM Conference on Global Computing Education, doi:10.1145/3300115.3309524 | |
| gi.citations.element | Joseph Psotka (2012): Distributed Technologies, In: Encyclopedia of the Sciences of Learning, doi:10.1007/978-1-4419-1428-6_987 | |
| gi.citations.element | Ruijia Cheng, Jenna Frens (2022): Feedback Exchange and Online Affinity: A Case Study of Online Fanfiction Writers, In: Proceedings of the ACM on Human-Computer Interaction CSCW2(6), doi:10.1145/3555127 | |
| gi.citations.element | Theresia Devi Indriasari, Paul Denny, Danielle Lottridge, Andrew Luxton-Reilly (2022): Gamification improves the quality of student peer code review, In: Computer Science Education 3(33), doi:10.1080/08993408.2022.2124094 | |
| gi.citations.element | Anbang Xu, Brian Bailey (2012): What do you think?, In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, doi:10.1145/2145204.2145252 | |
| gi.citations.element | Lina Battestilli, Apeksha Awasthi, Yingjun Cao (2018): Two-Stage Programming Projects, In: Proceedings of the 49th ACM Technical Symposium on Computer Science Education, doi:10.1145/3159450.3159486 | |
| gi.citations.element | Joe Gibbs Politz, Joseph M. Collard, Arjun Guha, Kathi Fisler, Shriram Krishnamurthi (2016): The Sweep, In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, doi:10.1145/2839509.2844626 | |
| gi.citations.element | Andrew Luxton-Reilly, Arthur Lewis, Beryl Plimmer (2018): Comparing sequential and parallel code review techniques for formative feedback, In: Proceedings of the 20th Australasian Computing Education Conference, doi:10.1145/3160489.3160498 | |
| gi.citations.element | Ruijia Cheng, Ziwen Zeng, Maysnow Liu, Steven Dow (2020): Critique Me, In: Proceedings of the ACM on Human-Computer Interaction CSCW2(4), doi:10.1145/3415232 | |
| gi.citations.element | Karin Väyrynen, Sonja Lutovac, Raimo Kaasila (2022): Reflection on peer reviewing as a pedagogical tool in higher education, In: Active Learning in Higher Education 3(24), doi:10.1177/14697874211073045 | |
| gi.citations.element | Christopher Hundhausen, Anukrati Agrawal, Dana Fairbrother, Michael Trevisan (2010): Does studio-based instruction work in CS 1?, In: Proceedings of the 41st ACM technical symposium on Computer science education, doi:10.1145/1734263.1734432 | |
| gi.citations.element | Nea Pirttinen, Juho Leinonen (2022): Can Students Review Their Peers?, In: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1, doi:10.1145/3502718.3524762 | |
| gi.citations.element | Christopher D. Hundhausen, Anukrati Agrawal, Pawan Agarwal (2013): Talking about code, In: ACM Transactions on Computing Education 3(13), doi:10.1145/2499947.2499951 | |
| gi.citations.element | Pawan Agarwal, Christopher D. Hundhausen (2010): A Socio-Psychological Approach to Improve Student Participation and Review Quality in Peer Code Reviews, In: 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, doi:10.1109/vlhcc.2010.50 | |
| gi.citations.element | Gabriel Badea, Elvira Popescu (2022): A Hybrid Approach for Mitigating Learners’ Rogue Review Behavior in Peer Assessment, In: Lecture Notes in Computer Science, doi:10.1007/978-3-031-09680-8_3 | |
| gi.citations.element | Sharique Hasan (2017): Conversational Peers and Idea Generation: Evidence from a Field Experiment, In: SSRN Electronic Journal, doi:10.2139/ssrn.2964214 | |
| gi.citations.element | Christopher Hundhausen, Anukrati Agrawal, Kyle Ryan (2010): The design of an online environment to support pedagogical code reviews, In: Proceedings of the 41st ACM technical symposium on Computer science education, doi:10.1145/1734263.1734324 | |
| gi.citations.element | Theresia Devi Indriasari, Andrew Luxton-Reilly, Paul Denny (2020): A Review of Peer Code Review in Higher Education, In: ACM Transactions on Computing Education 3(20), doi:10.1145/3403935 | |
| gi.citations.element | Diego Gudiño, María-Jesús Fernández-Sánchez, María-Teresa Becerra-Traver, Susana Sánchez-Herrera (2024): University Students’ Perceptions of Peer Assessment in Oral Presentations, In: Education Sciences 3(14), doi:10.3390/educsci14030221 | |
| gi.citations.element | Thomas Staubitz, Christoph Meinel (2020): A Systematic Quantitative and Qualitative Analysis of Participants’ Opinions on Peer Assessment in Surveys and Course Forum Discussions of MOOCs, In: 2020 IEEE Global Engineering Education Conference (EDUCON), doi:10.1109/educon45650.2020.9125089 | |
| gi.citations.element | Pantelis M. Papadopoulos, Antonio Cerone (2014): The Role of Peer Review in Supporting the Sustainability of Technology-Enhanced Learning Environments, In: Lecture Notes in Computer Science, doi:10.1007/978-3-642-54338-8_7 | |
| gi.citations.element | Muhammad Huzaifah Ismail, Shahrinaz Ismail, Muhammad Nur Affendy Nor'a, Jeffery Jeselee Sijore, Ainul Hazmin A Hamid (2024): A Systematic Literature Review on Recent Peer Code Review Implementation in Education, In: 2024 International Conference on TVET Excellence & Development (ICTeD), doi:10.1109/icted62334.2024.10844661 | |
| gi.citations.element | Meixiu Lu, Ming Ming Chiu (2022): Do Teamwork Guidelines Improve Peer Assessment Accuracy or Attitudes During Collaborative Learning?, In: IEEE Transactions on Education 4(65), doi:10.1109/te.2021.3130242 | |
| gi.citations.element | Dmytro Babik, Scott P. Stevens, Andrew Waters, David Tinapple (2020): The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study, In: IEEE Transactions on Learning Technologies 3(13), doi:10.1109/tlt.2020.2971495 | |
| gi.citations.element | Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh, Björn Hartmann (2012): CommunitySourcing, In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, doi:10.1145/2207676.2208619 | |
| gi.citations.element | Mireilla Bikanga Ada, Mohammad U. Majid (2022): Developing a system to increase motivation and engagement in student code peer review, In: 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), doi:10.1109/tale54877.2022.00023 | |
| gi.citations.element | Helen Wauck, Yu-Chun (Grace) Yen, Wai-Tat Fu, Elizabeth Gerber, Steven P. Dow, Brian P. Bailey (2017): From in the Class or in the Wild?, In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, doi:10.1145/3025453.3025477 | |
| gi.citations.element | Sharareh Alipour, Sina Elahimanesh, Soroush Jahanzad, Iman Mohammadi, Parimehr Morassafar, Seyed Parsa Neshaei, Mojtaba Tefagh (2024): Improving Grading Fairness and Transparency with Decentralized Collaborative Peer Assessment, In: Proceedings of the ACM on Human-Computer Interaction CSCW1(8), doi:10.1145/3637350 | |
| gi.citations.element | Thomas Staubitz, Dominic Petrick, Matthias Bauer, Jan Renz, Christoph Meinel (2016): Improving the Peer Assessment Experience on MOOC Platforms, In: Proceedings of the Third (2016) ACM Conference on Learning @ Scale, doi:10.1145/2876034.2876043 | |
| gi.citations.element | Veronika Dropčová, Zuzana Kubincová (2017): Team-Based Projects and Peer Assessment. IT Works!, In: Advances in Intelligent Systems and Computing, doi:10.1007/978-3-319-50337-0_10 | |
| gi.citations.element | Miki NAMATAME, Noboru MATSUDA (2016): Development of a Peer Review System for Art Education and its Evaluation, In: Transactions of Japan Society of Kansei Engineering 4(15), doi:10.5057/jjske.tjske-d-15-00091 | |
| gi.citations.element | Shanon Reckinger, Bryce Hughes (2020): Strategies for Implementing In-Class, Active, Programming Assessments, In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education, doi:10.1145/3328778.3366850 | |
| gi.citations.element | Amanda M. Holland-Minkley, Thomas Lombardi (2016): Improving Engagement in Introductory Courses with Homework Resubmission, In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, doi:10.1145/2839509.2844576 | |
| gi.citations.element | Dilrukshi Gamage, Mark E Whiting, Indika Perera, Shantha Fernando (2018): Improving Feedback and Discussion in MOOC Peer Assessement Using Introduced Peers, In: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), doi:10.1109/tale.2018.8615307 | |
| gi.citations.element | Li Meng, Zhou Xianzhong, Zhu Yingying (2010): An Evolutionary Game of Partners in Peer Review, In: 2010 Second World Congress on Software Engineering, doi:10.1109/wcse.2010.73 | |
| gi.citations.element | Igor S Montagner (2019): An experience with peer assessment in the context of a Computer Systems course, In: 2019 IEEE Frontiers in Education Conference (FIE), doi:10.1109/fie43999.2019.9028508 | |
| gi.citations.element | Celia Durkin, Federico Rossano, Scott Klemmer (2018): Score-Group Framing Negatively Impacts Peer Evaluations, In: Proceedings of the ACM on Human-Computer Interaction CSCW(2), doi:10.1145/3274314 | |
| gi.citations.element | Michael Brooks, Sumit Basu, Charles Jacobs, Lucy Vanderwende (2014): Divide and correct, In: Proceedings of the first ACM conference on Learning @ scale conference, doi:10.1145/2556325.2566243 | |
| gi.citations.element | Harald Søndergaard, Raoul A. Mulder (2012): Collaborative learning through formative peer review: pedagogy, programs and potential, In: Computer Science Education 4(22), doi:10.1080/08993408.2012.728041 | |
| gi.citations.element | Hans Yuan, Yingjun Cao (2019): Hybrid Pair Programming - A Promising Alternative to Standard Pair Programming, In: Proceedings of the 50th ACM Technical Symposium on Computer Science Education, doi:10.1145/3287324.3287352 | |
| gi.citations.element | Léon McGregor, Manuel Maarek (2020): Software Testing as Medium for Peer Feedback, In: United Kingdom & Ireland Computing Education Research conference., doi:10.1145/3416465.3416474 | |
| gi.conference.location | Sanibel Island, Florida, USA |