Item

Two Peers Are Better than One: Aggregating Peer Reviews for Computing Assignments is Surprisingly Accurate

Thumbnail Image

Fulltext URI

Document type

Additional Information

Date

2009

Authors

Journal Title

Journal ISSN

Volume Title

Source

Proceedings of the 2009 ACM International Conference on Supporting Group Work

Publisher

Association for Computing Machinery

Abstract

Scientific peer review, open source software development, wikis, and other domains use distributed review to improve quality of created content by providing feedback to the work's creator. Distributed review is used to assess or improve the quality of a work (e.g., an article). However, it can also provide learning benefits to the participants in the review process. We developed an online review system for beginning computer programming students; it gathers multiple anonymous peer reviews to give students feedback on their programming work. We deployed the system in an introductory programming class and evaluated it in a controlled study. We find that: peer reviews are accurate compared to an accepted evaluation standard, that students prefer reviews from other students with less experience than themselves, and that participating in a peer review process results in better learning outcomes.

Description

Reily, Ken; Finnerty, Pam Ludford; Terveen, Loren (2009): Two Peers Are Better than One: Aggregating Peer Reviews for Computing Assignments is Surprisingly Accurate. Proceedings of the 2009 ACM International Conference on Supporting Group Work. DOI: 10.1145/1531674.1531692. New York, NY, USA: Association for Computing Machinery. pp. 115–124. Sanibel Island, Florida, USA

Keywords

education, collaboration, peer review

Citation

DOI

10.1145/1531674.1531692

URI

URI

https://dl.eusset.eu/handle/20.500.12015/4862

Collections

GROUP 2009

Endorsement

Review

Supplemented By

Referenced By

Number of citations to item: 59

  • Sharareh Alipour, Sina Elahimanesh, Soroush Jahanzad, Iman Mohammadi, Parimehr Morassafar, Seyed Parsa Neshaei, Mojtaba Tefagh (2024): Improving Grading Fairness and Transparency with Decentralized Collaborative Peer Assessment, In: Proceedings of the ACM on Human-Computer Interaction CSCW1(8), doi:10.1145/3637350
  • Hans Yuan, Yingjun Cao (2019): Hybrid Pair Programming - A Promising Alternative to Standard Pair Programming, In: Proceedings of the 50th ACM Technical Symposium on Computer Science Education, doi:10.1145/3287324.3287352
  • Michael Brooks, Sumit Basu, Charles Jacobs, Lucy Vanderwende (2014): Divide and correct, In: Proceedings of the first ACM conference on Learning @ scale conference, doi:10.1145/2556325.2566243
  • Harald Søndergaard, Raoul A. Mulder (2012): Collaborative learning through formative peer review: pedagogy, programs and potential, In: Computer Science Education 4(22), doi:10.1080/08993408.2012.728041
  • Nea Pirttinen, Juho Leinonen (2022): Can Students Review Their Peers?, In: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1, doi:10.1145/3502718.3524762
  • Milorad Tosic, Valentina Nejkovic (2010): Trust-Based Peer Assessment for Virtual Learning Systems, In: Lecture Notes in Computer Science, doi:10.1007/978-3-642-16567-2_13
  • Ruijia Cheng, Ziwen Zeng, Maysnow Liu, Steven Dow (2020): Critique Me, In: Proceedings of the ACM on Human-Computer Interaction CSCW2(4), doi:10.1145/3415232
  • Andrew Luxton-Reilly, Arthur Lewis, Beryl Plimmer (2018): Comparing sequential and parallel code review techniques for formative feedback, In: Proceedings of the 20th Australasian Computing Education Conference, doi:10.1145/3160489.3160498
  • Anbang Xu, Brian Bailey (2012): What do you think?, In: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, doi:10.1145/2145204.2145252
  • Meixiu Lu, Ming Ming Chiu (2022): Do Teamwork Guidelines Improve Peer Assessment Accuracy or Attitudes During Collaborative Learning?, In: IEEE Transactions on Education 4(65), doi:10.1109/te.2021.3130242
  • Christopher Hundhausen, Anukrati Agrawal, Dana Fairbrother, Michael Trevisan (2010): Does studio-based instruction work in CS 1?, In: Proceedings of the 41st ACM technical symposium on Computer science education, doi:10.1145/1734263.1734432
  • Pawan Agarwal, Christopher D. Hundhausen (2010): A Socio-Psychological Approach to Improve Student Participation and Review Quality in Peer Code Reviews, In: 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, doi:10.1109/vlhcc.2010.50
  • Igor S Montagner (2019): An experience with peer assessment in the context of a Computer Systems course, In: 2019 IEEE Frontiers in Education Conference (FIE), doi:10.1109/fie43999.2019.9028508
  • Wilhelm A Friess, Andrew J Goupee (2019): Transformation of a Mechanical Engineering Capstone Experience, In: 2019 IEEE Frontiers in Education Conference (FIE), doi:10.1109/fie43999.2019.9028435
  • Gail L. Ring (2015): Implementing a Peer Mentoring Model in the Clemson ePortfolio Program, In: Theory Into Practice 4(54), doi:10.1080/00405841.2015.1077616
  • Anthony Estey, Jeremy Long, Bruce Gooch, Amy A. Gooch (2010): Investigating studio-based learning in a course on game design, In: Proceedings of the Fifth International Conference on the Foundations of Digital Games, doi:10.1145/1822348.1822357
  • Sandipan Sikdar, Matteo Marsili, Niloy Ganguly, Animesh Mukherjee (2017): Influence of Reviewer Interaction Network on Long-Term Citations: A Case Study of the Scientific Peer-Review System of the Journal of High Energy Physics, In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), doi:10.1109/jcdl.2017.7991572
  • Thomas Staubitz, Christoph Meinel (2020): A Systematic Quantitative and Qualitative Analysis of Participants’ Opinions on Peer Assessment in Surveys and Course Forum Discussions of MOOCs, In: 2020 IEEE Global Engineering Education Conference (EDUCON), doi:10.1109/educon45650.2020.9125089
  • Pantelis M. Papadopoulos, Thomas D. Lagkas, Stavros N. Demetriadis (2012): How to improve the peer review method: Free-selection vs assigned-pair protocol evaluated in a computer networking course, In: Computers & Education 2(59), doi:10.1016/j.compedu.2012.01.005
  • Dilrukshi Gamage, Mark E. Whiting, Thejan Rajapakshe, Haritha Thilakarathne, Indika Perera, Shantha Fernando (2017): Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives, In: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale, doi:10.1145/3051457.3054013
  • Veronika Dropčová, Zuzana Kubincová (2017): Team-Based Projects and Peer Assessment. IT Works!, In: Advances in Intelligent Systems and Computing, doi:10.1007/978-3-319-50337-0_10
  • Shanon Reckinger, Bryce Hughes (2020): Strategies for Implementing In-Class, Active, Programming Assessments, In: Proceedings of the 51st ACM Technical Symposium on Computer Science Education, doi:10.1145/3328778.3366850
  • Vivienne Farrell, Graham Farrell, Paul Kindler, Gilbert Ravalli, David Hall (2013): Capstone project online assessment tool without the paper work, In: Proceedings of the 18th ACM conference on Innovation and technology in computer science education, doi:10.1145/2462476.2462484
  • Axel Abels, Tom Lenaerts, Vito Trianni, Ann Nowé (2020): Collective Decision-Making as a Contextual Multi-armed Bandit Problem, In: Lecture Notes in Computer Science, doi:10.1007/978-3-030-63007-2_9
  • Theresia Devi Indriasari, Paul Denny, Danielle Lottridge, Andrew Luxton-Reilly (2022): Gamification improves the quality of student peer code review, In: Computer Science Education 3(33), doi:10.1080/08993408.2022.2124094
  • Amanda M. Holland-Minkley, Thomas Lombardi (2016): Improving Engagement in Introductory Courses with Homework Resubmission, In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, doi:10.1145/2839509.2844576
  • Juan Chen, Yingjun Cao, Linlin Du, Youwen Ouyang, Li Shen (2019): Improve Student Performance Using Moderated Two-Stage Projects, In: Proceedings of the ACM Conference on Global Computing Education, doi:10.1145/3300115.3309524
  • Christopher Hundhausen, Anukrati Agrawal, Kyle Ryan (2010): The design of an online environment to support pedagogical code reviews, In: Proceedings of the 41st ACM technical symposium on Computer science education, doi:10.1145/1734263.1734324
  • Diego Gudiño, María-Jesús Fernández-Sánchez, María-Teresa Becerra-Traver, Susana Sánchez-Herrera (2024): University Students’ Perceptions of Peer Assessment in Oral Presentations, In: Education Sciences 3(14), doi:10.3390/educsci14030221
  • Joe Gibbs Politz, Joseph M. Collard, Arjun Guha, Kathi Fisler, Shriram Krishnamurthi (2016): The Sweep, In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, doi:10.1145/2839509.2844626
  • Ruijia Cheng, Jenna Frens (2022): Feedback Exchange and Online Affinity: A Case Study of Online Fanfiction Writers, In: Proceedings of the ACM on Human-Computer Interaction CSCW2(6), doi:10.1145/3555127
  • Axel Abels, Tom Lenaerts, Vito Trianni, Ann Nowé (2020): How Expert Confidence Can Improve Collective Decision-Making in Contextual Multi-Armed Bandit Problems, In: Lecture Notes in Computer Science, doi:10.1007/978-3-030-63007-2_10
  • Daniel Olivares, Christopher Hundhausen, Namrata Ray (2021): Designing IDE Interventions to Promote Social Interaction and Improved Programming Outcomes in Early Computing Courses, In: ACM Transactions on Computing Education 1(22), doi:10.1145/3453165
  • Yubo Kou, Colin M. Gray (2017): Supporting Distributed Critique through Interpretation and Sense-Making in an Online Creative Community, In: Proceedings of the ACM on Human-Computer Interaction CSCW(1), doi:10.1145/3134695
  • Li Meng, Zhou Xianzhong, Zhu Yingying (2010): An Evolutionary Game of Partners in Peer Review, In: 2010 Second World Congress on Software Engineering, doi:10.1109/wcse.2010.73
  • Dilrukshi Gamage, Mark E Whiting, Indika Perera, Shantha Fernando (2018): Improving Feedback and Discussion in MOOC Peer Assessement Using Introduced Peers, In: 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), doi:10.1109/tale.2018.8615307
  • Mireilla Bikanga Ada, Mohammad U. Majid (2022): Developing a system to increase motivation and engagement in student code peer review, In: 2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), doi:10.1109/tale54877.2022.00023
  • Sharique Hasan, Rembrand Koning (2019): Conversations and idea generation: Evidence from a field experiment, In: Research Policy 9(48), doi:10.1016/j.respol.2019.103811
  • Celia Durkin, Federico Rossano, Scott Klemmer (2018): Score-Group Framing Negatively Impacts Peer Evaluations, In: Proceedings of the ACM on Human-Computer Interaction CSCW(2), doi:10.1145/3274314
  • Pantelis M. Papadopoulos, Antonio Cerone (2014): The Role of Peer Review in Supporting the Sustainability of Technology-Enhanced Learning Environments, In: Lecture Notes in Computer Science, doi:10.1007/978-3-642-54338-8_7
  • Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh, Björn Hartmann (2012): CommunitySourcing, In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, doi:10.1145/2207676.2208619
  • Thomas Staubitz, Dominic Petrick, Matthias Bauer, Jan Renz, Christoph Meinel (2016): Improving the Peer Assessment Experience on MOOC Platforms, In: Proceedings of the Third (2016) ACM Conference on Learning @ Scale, doi:10.1145/2876034.2876043
  • Miki NAMATAME, Noboru MATSUDA (2016): Development of a Peer Review System for Art Education and its Evaluation, In: Transactions of Japan Society of Kansei Engineering 4(15), doi:10.5057/jjske.tjske-d-15-00091
  • Karin Väyrynen, Sonja Lutovac, Raimo Kaasila (2022): Reflection on peer reviewing as a pedagogical tool in higher education, In: Active Learning in Higher Education 3(24), doi:10.1177/14697874211073045
  • Lina Battestilli, Apeksha Awasthi, Yingjun Cao (2018): Two-Stage Programming Projects, In: Proceedings of the 49th ACM Technical Symposium on Computer Science Education, doi:10.1145/3159450.3159486
  • Dmytro Babik, Scott P. Stevens, Andrew Waters, David Tinapple (2020): The Effects of Dispersion and Reciprocity on Assessment Fidelity in Peer-Review Systems: A Simulation Study, In: IEEE Transactions on Learning Technologies 3(13), doi:10.1109/tlt.2020.2971495
  • Joe Gibbs Politz, Shriram Krishnamurthi, Kathi Fisler (2014): In-flow peer-review of tests in test-first programming, In: Proceedings of the tenth annual conference on International computing education research, doi:10.1145/2632320.2632347
  • Zuzana Kubincová, Martin Homola, Veronika Dropčová (2017): Peer Reviews in a Web Design Course: Now Students like Them Too, In: Lecture Notes in Computer Science, doi:10.1007/978-3-319-52836-6_66
  • Tomás Alves, Francisco Sousa, Sandra Gama, Joaquim Jorge, Daniel Gonçalves (2023): How Personality Traits Affect Peer Assessment in Distance Learning, In: Technology, Knowledge and Learning 1(29), doi:10.1007/s10758-023-09694-2
  • Christopher D. Hundhausen, Pawan Agarwal, Michael Trevisan (2011): Online vs. face-to-face pedagogical code reviews, In: Proceedings of the 42nd ACM technical symposium on Computer science education, doi:10.1145/1953163.1953201
  • Dave Clarke, Tony Clear, Kathi Fisler, Matthias Hauswirth, Shriram Krishnamurthi, Joe Gibbs Politz, Ville Tirronen, Tobias Wrigstad (2014): In-Flow Peer Review, In: Proceedings of the Working Group Reports of the 2014 on Innovation & Technology in Computer Science Education Conference, doi:10.1145/2713609.2713612
  • Ville Hyyrynen, Harri Hämäläinen, Jouni Ikonen, Jari Porras (2010): MyPeerReview, In: Proceedings of the 10th Koli Calling International Conference on Computing Education Research, doi:10.1145/1930464.1930481
  • Christopher D. Hundhausen, Anukrati Agrawal, Pawan Agarwal (2013): Talking about code, In: ACM Transactions on Computing Education 3(13), doi:10.1145/2499947.2499951
  • Gabriel Badea, Elvira Popescu (2022): A Hybrid Approach for Mitigating Learners’ Rogue Review Behavior in Peer Assessment, In: Lecture Notes in Computer Science, doi:10.1007/978-3-031-09680-8_3
  • Sharique Hasan (2017): Conversational Peers and Idea Generation: Evidence from a Field Experiment, In: SSRN Electronic Journal, doi:10.2139/ssrn.2964214
  • Léon McGregor, Manuel Maarek (2020): Software Testing as Medium for Peer Feedback, In: United Kingdom & Ireland Computing Education Research conference., doi:10.1145/3416465.3416474
  • Joseph Psotka (2012): Distributed Technologies, In: Encyclopedia of the Sciences of Learning, doi:10.1007/978-1-4419-1428-6_987
  • Helen Wauck, Yu-Chun (Grace) Yen, Wai-Tat Fu, Elizabeth Gerber, Steven P. Dow, Brian P. Bailey (2017): From in the Class or in the Wild?, In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, doi:10.1145/3025453.3025477
  • Theresia Devi Indriasari, Andrew Luxton-Reilly, Paul Denny (2020): A Review of Peer Code Review in Higher Education, In: ACM Transactions on Computing Education 3(20), doi:10.1145/3403935
Please note: Providing information about citations is only possible thanks to to the open metadata APIs provided by crossref.org and opencitations.net. These lists may be incomplete due to unavailable citation data.source: opencitations.net, crossref.org