The Problem of Majority Voting in Crowdsourcing with Binary Classes

dc.contributor.authorSalminen, Joni
dc.contributor.authorKamel, Ahmed Mohamed
dc.contributor.authorJung, Soon-Gyo
dc.contributor.authorJansen, Bernard
dc.date.accessioned2021-05-18T10:05:05Z
dc.date.available2021-05-18T10:05:05Z
dc.date.issued2021
dc.description.abstractWhen there are two classes, a majority vote can always be obtained with three labelers. Researchers can utilize this property to obtain a false sense of confidence in their ground truth labels. We demonstrate such a case with 3000 crowdsourced labels for an online hate dataset. Evaluating with percentage agreement, Gwet’s AC1, and Krippendorff’s alpha, results show that using more raters teases out the hidden nuances in raters’ preferences. We show that full agreement among the raters monotonically decreases from three raters (28.4%) to nine raters (19.5%). Ten raters have a higher agreement than any other number of raters, which supports the idea of increasing the number of raters for subjective labeling tasks. Nevertheless, while beneficial, increasing the number of raters cannot be considered as a fundamental solution to the issue of agreement in subjective crowdsourcing tasks, as even with ten raters, there is a non- negligible number of ties (4.11%). We suggest having a small sample of the data labeled by five or more raters to evaluate the stability of agreement among the raters.en
dc.identifier.doi10.18420/ecscw2021_n12
dc.identifier.pissn2510-2591
dc.identifier.urihttps://dl.eusset.eu/handle/20.500.12015/4163
dc.language.isoen
dc.publisherEuropean Society for Socially Embedded Technologies (EUSSET)
dc.relation.ispartofProceedings of 19th European Conference on Computer-Supported Cooperative Work
dc.relation.ispartofseriesECSCW
dc.relation.ispartofseriesECSCW
dc.titleThe Problem of Majority Voting in Crowdsourcing with Binary Classesen
dc.typeText/Conference Paper
gi.conference.date7-11 June 2021
gi.conference.locationZurich, Switzerland
gi.conference.sessiontitleNotes
mci.conference.reviewfull

Files

Original bundle
1 - 1 of 1
Loading...
Thumbnail Image
Name:
ecscw2021-n12.pdf
Size:
501.55 KB
Format:
Adobe Portable Document Format