Please use this identifier to cite or link to this item: https://dl.eusset.eu/handle/20.500.12015/4163
Title: The Problem of Majority Voting in Crowdsourcing with Binary Classes
Authors: Salminen, Joni
Kamel, Ahmed Mohamed
Jung, Soon-Gyo
Jansen, Bernard
Issue Date: 2021
Publisher: European Society for Socially Embedded Technologies (EUSSET)
metadata.dc.relation.ispartof: Proceedings of 19th European Conference on Computer-Supported Cooperative Work
Series/Report no.: ECSCW
ECSCW
Abstract: When there are two classes, a majority vote can always be obtained with three labelers. Researchers can utilize this property to obtain a false sense of confidence in their ground truth labels. We demonstrate such a case with 3000 crowdsourced labels for an online hate dataset. Evaluating with percentage agreement, Gwet’s AC1, and Krippendorff’s alpha, results show that using more raters teases out the hidden nuances in raters’ preferences. We show that full agreement among the raters monotonically decreases from three raters (28.4%) to nine raters (19.5%). Ten raters have a higher agreement than any other number of raters, which supports the idea of increasing the number of raters for subjective labeling tasks. Nevertheless, while beneficial, increasing the number of raters cannot be considered as a fundamental solution to the issue of agreement in subjective crowdsourcing tasks, as even with ten raters, there is a non- negligible number of ties (4.11%). We suggest having a small sample of the data labeled by five or more raters to evaluate the stability of agreement among the raters.
metadata.dc.identifier.doi: 10.18420/ecscw2021_n12
URI: https://dl.eusset.eu/handle/20.500.12015/4163
ISSN: 2510-2591
metadata.mci.conference.sessiontitle: Notes
metadata.mci.conference.location: Zurich, Switzerland
metadata.mci.conference.date: 7-11 June 2021
Appears in Collections:ECSCW 2021 Exploratory Papers and Notes

Files in This Item:
File SizeFormat 
ecscw2021-n12.pdf501,55 kBAdobe PDFView/Open


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.