Please use this identifier to cite or link to this item: https://dl.eusset.eu/handle/20.500.12015/3746
Title: Crowd Anatomy Beyond the Good and Bad: Behavioral Traces for Crowd Worker Modeling and Pre-selection
Authors: Gadiraju, Ujwal
Demartini, Gianluca
Kawase, Ricardo
Dietze, Stefan
Keywords: Behavioral traces;Crowdsourcing;Microtasks;Pre-screening;Pre-selection;Worker typology;Workers
Issue Date: 2019
Publisher: Springer
metadata.dc.relation.ispartof: Computer Supported Cooperative Work (CSCW): Vol. 28, No. 5
metadata.mci.reference.pages: 815-841
Series/Report no.: Computer Supported Cooperative Work (CSCW)
Abstract: The suitability of crowdsourcing to solve a variety of problems has been investigated widely. Yet, there is still a lack of understanding about the distinct behavior and performance of workers within microtasks. In this paper, we first introduce a fine-grained data-driven worker typology based on different dimensions and derived from behavioral traces of workers. Next, we propose and evaluate novel models of crowd worker behavior and show the benefits of behavior-based worker pre-selection using machine learning models. We also study the effect of task complexity on worker behavior. Finally, we evaluate our novel typology-based worker pre-selection method in image transcription and information finding tasks involving crowd workers completing 1,800 HITs. Our proposed method for worker pre-selection leads to a higher quality of results when compared to the standard practice of using qualification or pre-screening tests. For image transcription tasks our method resulted in an accuracy increase of nearly 7% over the baseline and of almost 10% in information finding tasks, without a significant difference in task completion time. Our findings have important implications for crowdsourcing systems where a worker’s behavioral type is unknown prior to participation in a task. We highlight the potential of leveraging worker types to identify and aid those workers who require further training to improve their performance. Having proposed a powerful automated mechanism to detect worker types, we reflect on promoting fairness, trust and transparency in microtask crowdsourcing platforms.
metadata.dc.identifier.doi: 10.1007/s10606-018-9336-y
URI: http://dx.doi.org/10.1007/s10606-018-9336-y
https://dl.eusset.eu/handle/20.500.12015/3746
ISSN: 1573-7551
Appears in Collections:JCSCW Vol. 28 (2019)

Files in This Item:
There are no files associated with this item.


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.