Publications
Publications in reversed chronological order. Please check my Google Scholar page for more details.
2024
- Characterizing Information Seeking Processes with Multiple Physiological SignalsIn Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24), 2024To appear.
- Walert: Putting Conversational Information Seeking Knowledge into Action by Building and Evaluating a Large Language Model-Powered ChatbotSachin Pathiyan Cherumanal, Lin Tian, Futoon M. Abushaqra, Angel Felipe Paula, Kaixin Ji, Danula Hettiachchi, Johanne R. Trippas, Halil Ali, Falk Scholer, and Damiano SpinaIn Proceedings of ACM SIGIR Conference on Human Information Interaction And Retrieval (CHIIR ’24), 2024
Creating and deploying customized applications is crucial for operational success and enriching user experiences in the rapidly evolving modern business world. A prominent facet of modern user experiences is the integration of chatbots or voice assistants. The rapid evolution of Large Language Models (LLMs) has provided a powerful tool to build conversational applications. We present Walert, a customized LLM-based conversational agent able to answer frequently asked questions about computer science degrees and programs at RMIT University. Our demo aims to showcase how conversational information-seeking researchers can effectively communicate the benefits of using best practices to stakeholders interested in developing and deploying LLM-based chatbots. These practices are well-known in our community but often overlooked by practitioners who may not have access to this knowledge. The methodology and resources used in this demo serve as a bridge to facilitate knowledge transfer from experts, address industry professionals’ practical needs, and foster a collaborative environment. The data and code of the demo are available at https://github.com/rmit-ir/walert.
@inproceedings{Pathiyan2024-wp, author = {Pathiyan Cherumanal, Sachin and Tian, Lin and Abushaqra, Futoon M. and Magnossão de Paula, Angel Felipe and Ji, Kaixin and Hettiachchi, Danula and Trippas, Johanne R. and Ali, Halil and Scholer, Falk and Spina, Damiano}, booktitle = {Proceedings of ACM SIGIR Conference on Human Information Interaction And Retrieval}, series = {CHIIR '24}, title = {Walert: Putting Conversational Information Seeking Knowledge into Action by Building and Evaluating a Large Language Model-Powered Chatbot}, year = {2024}, publisher = {ACM}, doi = {https://doi.org/10.1145/3627508.3638309}, }
2023
- Designing and Evaluating Presentation Strategies for Fact-Checked ContentDanula Hettiachchi, Kaixin Ji, Jenny Kennedy, Anthony McCosker, Flora D Salim, Mark Sanderson, Falk Scholer, and Damiano SpinaIn Proceedings of the 32nd ACM International Conference on Information & Knowledge Management (CIKM ’23), 2023
With the rapid growth of online misinformation, it is crucial to have reliable fact-checking methods. Recent research on finding check-worthy claims and automated fact-checking have made significant advancements. However, limited guidance exists regarding the presentation of fact-checked content to effectively convey verified information to users. We address this research gap by exploring the critical design elements in fact-checking reports and investigating whether credibility and presentation-based design improvements can enhance users’ ability to interpret the report accurately. We co-developed potential content presentation strategies through a workshop involving fact-checking professionals, communication experts, and researchers. The workshop examined the significance and utility of elements such as veracity indicators and explored the feasibility of incorporating interactive components for enhanced information disclosure. Building on the workshop outcomes, we conducted an online experiment involving 76 crowd workers to assess the efficacy of different design strategies. The results indicate that proposed strategies significantly improve users’ ability to accurately interpret the verdict of fact-checking articles. Our findings underscore the critical role of effective presentation of fact reports in addressing the spread of misinformation. By adopting appropriate design enhancements, the effectiveness of fact-checking reports can be maximized, enabling users to make informed judgments.
@inproceedings{Hettiachchi2023-po, title = {Designing and Evaluating Presentation Strategies for {Fact-Checked} Content}, booktitle = {Proceedings of the 32nd {ACM} International Conference on Information \& Knowledge Management}, author = {Hettiachchi, Danula and Ji, Kaixin and Kennedy, Jenny and McCosker, Anthony and Salim, Flora D and Sanderson, Mark and Scholer, Falk and Spina, Damiano}, publisher = {ACM}, series = {CIKM '23}, year = {2023}, doi = {10.1145/3583780.3614841}, } - How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in TweetsDanula Hettiachchi, Indigo Holcombe-James, Stephanie Livingstone, Anjalee Silva, Matthew Lease, Flora D Salim, and Mark SandersonIn Proceedings of the Eleventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP ’23), 2023
Crowdsourced annotation is vital to both collecting labelled data to train and test automated content moderation systems and to support human-in-the-loop review of system decisions. However, annotation tasks such as judging hate speech are subjective and thus highly sensitive to biases stemming from annotator beliefs, characteristics and demographics. We conduct two crowdsourcing studies on Mechanical Turk to examine annotator bias in labelling sexist and misogynistic hate speech. Results from 109 annotators show that annotator political inclination, moral integrity, personality traits, and sexist attitudes significantly impact annotation accuracy and the tendency to tag content as hate speech. In addition, semi-structured interviews with nine crowd workers provide further insights regarding the influence of subjectivity on annotations. In exploring how workers interpret a task - shaped by complex negotiations between platform structures, task instructions, subjective motivations, and external contextual factors - we see annotations not only impacted by worker factors but also simultaneously shaped by the structures under which they labour.
@inproceedings{Hettiachchi2023-hj, title = {How Crowd Worker Factors Influence Subjective Annotations: A Study of Tagging Misogynistic Hate Speech in Tweets}, booktitle = {Proceedings of the Eleventh {AAAI} Conference on Human Computation and Crowdsourcing}, author = {Hettiachchi, Danula and Holcombe-James, Indigo and Livingstone, Stephanie and de Silva, Anjalee and Lease, Matthew and Salim, Flora D and Sanderson, Mark}, series = {HCOMP '23}, volume = {11}, number = {1}, year = {2023}, publisher = {AAAI}, doi = {10.1609/hcomp.v11i1.27546}, dimensions = {true} } - Are footpaths encroached by shared e-scooters? Spatio-temporal Analysis of Micro-mobility ServicesIn 2023 24th IEEE International Conference on Mobile Data Management (MDM ’23), 2023
Micro-mobility services (e.g., e-bikes, e-scooters) are increasingly popular among urban communities, being a flexible transport option that brings both opportunities and challenges. As a growing mode of transportation, insights gained from micro-mobility usage data are valuable in policy formulation and improving the quality of services. Existing research analyses patterns and features associated with usage distributions in different localities, and focuses on either temporal or spatial aspects. In this paper, we employ a combination of methods that analyse both spatial and temporal characteristics related to e-scooter trips in a more granular level, enabling observations at different time frames and local geographical zones that prior analysis wasn’t able to do. The insights obtained from anonymised, restricted data on shared e-scooter rides show the applicability of the employed method on regulated, privacy preserving micro-mobility trip data. Our results showed population density is the topmost important feature, and it associates with e-scooter usage positively. Population owning motor vehicles is negatively associated with shared e-scooter trips, suggesting a reduction in e-scooter usage among motor vehicle owners. Furthermore, we found that the effect of humidity is more important than precipitation in predicting hourly e-scooter trip count. Buffer analysis showed, nearly 29% trips were stopped, and 27% trips were started on the footpath, revealing higher utilisation of footpaths for parking e-scooters in Melbourne.
@inproceedings{Kegalle2023-ux, title = {Are footpaths encroached by shared e-scooters? Spatio-temporal Analysis of Micro-mobility Services}, booktitle = {2023 24th {IEEE} International Conference on Mobile Data Management}, author = {Kegalle, Hiruni and Hettiachchi, Danula and Chan, Jeffrey and Salim, Flora and Sanderson, Mark}, series = {MDM '23}, pages = {255--264}, publisher = {IEEE}, year = {2023}, issn = {2375-0324}, doi = {10.1109/MDM58254.2023.00049}, } - Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing ActivitiesIn Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’23), 2023
Physiological signals can potentially be applied as objective measures to understand the behavior and engagement of users interacting with information access systems. However, the signals are highly sensitive, and many controls are required in laboratory user studies. To investigate the extent to which controlled or uncontrolled (i.e., confounding) variables such as task sequence or duration influence the observed signals, we conducted a pilot study where each participant completed four types of information-processing activities (READ, LISTEN, SPEAK, and WRITE). Meanwhile, we collected data on blood volume pulse, electrodermal activity, and pupil responses. We then used machine learning approaches as a mechanism to examine the influence of controlled and uncontrolled variables that commonly arise in user studies. Task duration was found to have a substantial effect on the model performance, suggesting it represents individual differences rather than giving insight into the target variables. This work contributes to our understanding of such variables in using physiological signals in information retrieval user studies.
@inproceedings{Ji2023-aq, title = {Examining the Impact of Uncontrolled Variables on Physiological Signals in User Studies for Information Processing Activities}, booktitle = {Proceedings of the 46th International {ACM} {SIGIR} Conference on Research and Development in Information Retrieval}, author = {Ji, Kaixin and Spina, Damiano and Hettiachchi, Danula and Salim, Flora D and Scholer, Falk}, publisher = {ACM}, series = {SIGIR '23}, year = {2023}, doi = {10.1145/3539618.3591981}, } - Combining Worker Factors for Heterogeneous Crowd Task AssignmentSenuri Wijenayake, Danula Hettiachchi, and Jorge GoncalvesIn Proceedings of the ACM Web Conference 2023 (WWW ’23), 2023
Optimising the assignment of tasks to workers is an effective approach to ensure high quality in crowdsourced data - particularly in heterogeneous micro tasks. However, previous attempts at heterogeneous micro task assignment based on worker characteristics are limited to using cognitive skills, despite literature emphasising that worker performance varies based on other parameters. This study is an initial step towards understanding whether and how multiple parameters such as cognitive skills, mood, personality, alertness, comprehension skill, and social and physical context of workers can be leveraged in tandem to improve worker performance estimations in heterogeneous micro tasks. Our predictive models indicate that these parameters have varying effects on worker performance in the five task types considered – sentiment analysis, classification, transcription, named entity recognition and bounding box. Moreover, we note 0.003 - 0.018 reduction in mean absolute error of predicted worker accuracy across all tasks, when task assignment is based on models that consider all parameters vs. models that only consider workers’ cognitive skills. Our findings pave the way for the use of holistic approaches in micro task assignment that effectively quantify worker context.
@inproceedings{Wijenayake2023-yc, title = {Combining Worker Factors for Heterogeneous Crowd Task Assignment}, booktitle = {Proceedings of the {ACM} Web Conference 2023}, author = {Wijenayake, Senuri and Hettiachchi, Danula and Goncalves, Jorge}, publisher = {Association for Computing Machinery}, pages = {3794--3805}, series = {WWW '23}, year = {2023}, address = {New York, NY, USA}, location = {Austin, TX, USA}, isbn = {9781450394161}, doi = {10.1145/3543507.3583190}, talk = {https://youtu.be/cZoBigEPAi4}, } - Mapping 20 years of accessibility research in HCI: A co-word analysisZhanna Sarsenbayeva, Niels van Berkel, Danula Hettiachchi, Benjamin Tag, Eduardo Velloso, Jorge Goncalves, and Vassilis KostakosInternational Journal of Human Computer Studies, 175(), 103018, 2023
We employ hierarchical clustering, strategic diagrams, and network core–periphery analysis to assess and visualise the intellectual progress of accessibility research within HCI in the past two decades. The study quantifies and explains the development of accessibility research and its thematic evolution based on 1,535 papers published at TACCESS, ASSETS, IJHCS, and CHI and their respective 3,470 author-assigned keywords. The novelty of this work is based on employing a quantitative methodological approach to provide an overview of accessibility research progress and insights into its driving and trending themes through the period 2001–2021. In addition, we identify declining, emerging, and core backbone themes of accessibility research. Finally, we discuss the opportunities for research that arise from our findings. These contributions provide a roadmap for researchers working on accessibility.
@article{Sarsenbayeva2023-mo, title = {Mapping 20 years of accessibility research in {HCI}: A co-word analysis}, author = {Sarsenbayeva, Zhanna and {van Berkel}, Niels and Hettiachchi, Danula and Tag, Benjamin and Velloso, Eduardo and Goncalves, Jorge and Kostakos, Vassilis}, journal = {International Journal of Human Computer Studies}, volume = {175}, pages = {103018}, year = {2023}, issn = {1071-5819}, doi = {10.1016/j.ijhcs.2023.103018}, } - RMIT_IR at the NTCIR-17 FairWeb-1 TaskSachin Pathiyan Cherumanal, Kaixin Ji, Danula Hettiachchi, Johanne R Trippas, Falk Scholer, and Damiano SpinaIn Proceedings of 17th Conference on Evaluation of Information Access Technologies (NTCIR-17), 2023
This report describes the participation of the RMIT IR group at the NTCIR-17 FairWeb-1 task. We submitted five runs with the aim of exploring the role of explicit search result diversification (SRD) and ranking fusion to generate fair rankings considering multiple fairness attributes. We also explored the use of a linear combination-based technique (LC) to take into consideration the relevance while re-ranking. In this report, we compared results from all our submitted runs against each other and the retrieval baselines along each topic type separately (i.e., Researcher, Movie, YouTube). Overall, our results show that neither the SRD-based runs nor the linear combination-based runs show any statistically significant improvement over the retrieval baselines. The source code of the framework for generating group memberships is made available at https://github.com/rmit-ir/fairweb-1/
@inproceedings{Cherumanal2023-or, title = {{RMIT\_IR} at the {NTCIR-17} {FairWeb-1} Task}, booktitle = {Proceedings of 17th Conference on Evaluation of Information Access Technologies}, author = {Cherumanal, Sachin Pathiyan and Ji, Kaixin and Hettiachchi, Danula and Trippas, Johanne R and Scholer, Falk and Spina, Damiano}, publisher = {National Institute of Informatics}, series = {NTCIR-17}, year = {2023}, address = {Tokyo, Japan}, doi = {10.20736/0002001315}, } - Towards Detecting Tonic Information Processing Activities with Physiological DataIn Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing (UbiComp/ISWC ’23 Adjunct), 2023
Characterizing Information Processing Activities (IPAs) such as reading, listening, speaking, and writing, with physiological signals captured by wearable sensors can broaden the understanding of how people produce and consume information. However, sensors are highly sensitive to external conditions that are not trivial to control – not even in lab user studies. We conducted a pilot study (N = 7) to assess the robustness and sensitivity of physiological signals across four IPAs (READ, LISTEN, SPEAK, and WRITE) using multiple sensors. The collected signals include Electrodermal Activities, Blood Volume Pulse, gaze, and head motion. We observed consistent trends across participants, and ten features with statistically significant differences across the four IPAs. Our results provide preliminary quantitative evidence of differences in physiological responses when users encounter IPAs, revealing the necessity to inspect the signals separately according to the IPAs. The next step of this study moves into a specific context, information retrieval, and the IPAs are considered as the interaction modalities with the search system, for instance, submitting the search query by speaking or typing.
@inproceedings{Ji2023-td, author = {Ji, Kaixin and Spina, Damiano and Hettiachchi, Danula and Scholer, Falk and Salim, Flora D.}, title = {Towards Detecting Tonic Information Processing Activities with Physiological Data}, year = {2023}, isbn = {9798400702006}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, url = {https://doi.org/10.1145/3594739.3610679}, doi = {10.1145/3594739.3610679}, booktitle = {Adjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing \& the 2023 ACM International Symposium on Wearable Computing}, pages = {1–5}, numpages = {5}, location = {Cancun, Mexico}, series = {UbiComp/ISWC '23 Adjunct}, } - Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence ExplanationsEdward Small, Yueqing Xuan, Danula Hettiachchi, and Kacper SokolIn ACM CHI Workshop Human-Centered Perspectives in Explainable AI, 2023
Explainable artificial intelligence techniques are developed at breakneck speed, but suitable evaluation approaches lag behind. With explainers becoming increasingly complex and a lack of consensus on how to assess their utility, it is challenging to judge the benefit and effectiveness of different explanations. To address this gap, we take a step back from sophisticated predictive algorithms and instead look into explainability of simple decision-making models. In this setting, we aim to assess how people perceive comprehensibility of their different representations such as mathematical formulation, graphical representation and textual summarisation (of varying complexity and scope). This allows us to capture how diverse stakeholders – engineers, researchers, consumers, regulators and the like – judge intelligibility of fundamental concepts that more elaborate artificial intelligence explanations are built from. This position paper charts our approach to establishing appropriate evaluation methodology as well as a conceptual and practical framework to facilitate setting up and executing relevant user studies.
@inproceedings{Small2023-by, title = {Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations}, booktitle = {{ACM} {CHI} Workshop {Human-Centered} Perspectives in Explainable {AI}}, author = {Small, Edward and Xuan, Yueqing and Hettiachchi, Danula and Sokol, Kacper}, month = mar, year = {2023}, } - Towards Measuring Sensitivity of Psychometrics in Crowdsourcing Tasks: Engaging with Fact-checked Content OnlineStanislaus Krisna, Danula Hettiachchi, and Damiano SpinaIn Proceedings of the CSCW 2023 Workshop on Undertanding and Mitigating Cognitive Biases in Human-AI Collaboration (CSCW ’23 Workshop), 2023
This study attempts to explore the relationships between psychometric test outcomes and how participants rate the truthfulness of online news articles. We seek to be able to help in the identification of misinformation. We attempt to explore the relationship between political orientation and several other psychometrics. We then ask the user to rate social media article as well as fact-checked article before finding the relationship between the variables and the difference in rating of each articles. We faced an issue in aggregating the result of each participant as well as computing the distance in ordered scale (magnitude given values in limited range).
@inproceedings{Krisna2023-tm, author = {Krisna, Stanislaus and Hettiachchi, Danula and Spina, Damiano}, booktitle = {Proceedings of the CSCW 2023 Workshop on Undertanding and Mitigating Cognitive Biases in Human-AI Collaboration}, pages = {4}, title = {Towards Measuring Sensitivity of Psychometrics in Crowdsourcing Tasks: Engaging with Fact-checked Content Online}, year = {2023}, series = {CSCW '23 Workshop}, talk = {https://www.damianospina.com/publication/krisna-2023-towards/krisna-2023-towards.mp4}, }
2022
- A Survey on Task Assignment in CrowdsourcingDanula Hettiachchi, Vassilis Kostakos, and Jorge GoncalvesACM Computing Surveys, 55(3), 1–35, 2022
Quality improvement methods are essential to gathering high-quality crowdsourced data, both for research and industry applications. A popular and broadly applicable method is task assignment that dynamically adjusts crowd workflow parameters. In this survey, we review task assignment methods that address: heterogeneous task assignment, question assignment, and plurality problems in crowdsourcing. We discuss and contrast how these methods estimate worker performance, and highlight potential challenges in their implementation. Finally, we discuss future research directions for task assignment methods, and how crowdsourcing platforms and other stakeholders can benefit from them.
@article{Hettiachchi2022-vq, title = {A Survey on Task Assignment in Crowdsourcing}, author = {Hettiachchi, Danula and Kostakos, Vassilis and Goncalves, Jorge}, journal = {ACM Computing Surveys}, publisher = {ACM}, volume = {55}, number = {3}, pages = {1--35}, month = feb, year = {2022}, address = {New York, NY, USA}, issn = {0360-0300}, doi = {10.1145/3494522}, } - Does a Face Mask Protect My Privacy?: Deep Learning to Predict Protected Attributes from Masked Face ImagesSachith Seneviratne, Nuran Kasthuriarachchi, Sanka Rasnayaka, Danula Hettiachchi, and Ridwan ShariffdeenIn AI 2021: Advances in Artificial Intelligence (AJCAI ’21), 2022
Contactless and efficient systems are implemented rapidly to advocate preventive methods in the fight against the COVID-19 pandemic. Despite the positive benefits of such systems, there is potential for exploitation by invading user privacy. In this work, we analyse the privacy invasiveness of face biometric systems by predicting privacy-sensitive soft-biometrics using masked face images. We train and apply a CNN based on the ResNet-50 architecture with 20,003 synthetic masked images and measure the privacy invasiveness. Despite the popular belief of the privacy benefits of wearing a mask among people, we show that there is no significant difference to privacy invasiveness when a mask is worn. In our experiments we were able to accurately predict sex (94.7%), race (83.1%) and age (MAE 6.21 and RMSE 8.33) from masked face images. Our proposed approach can serve as a baseline utility to evaluate the privacy-invasiveness of artificial intelligence systems that make use of privacy-sensitive information. We open-source all contributions for reproducibility and broader use by the research community.
@inproceedings{Seneviratne2022-ll, title = {Does a Face Mask Protect My Privacy?: Deep Learning to Predict Protected Attributes from Masked Face Images}, booktitle = {{AI} 2021: Advances in Artificial Intelligence}, author = {Seneviratne, Sachith and Kasthuriarachchi, Nuran and Rasnayaka, Sanka and Hettiachchi, Danula and Shariffdeen, Ridwan}, publisher = {Springer}, pages = {91--102}, year = {2022}, series = {AJCAI '21}, doi = {10.1007/978-3-030-97546-3\_8}, } - REGROW: Reimagining Global Crowdsourcing for Better Human-AI CollaborationAndy Alorwu, Saiph Savage, Niels van Berkel, Dmitry Ustalov, Alexey Drutsa, Jonas Oppenlaender, Oliver Bates, Danula Hettiachchi, Ujwal Gadiraju, Jorge Goncalves, and Simo HosioIn Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA ’22), 2022
Crowdworkers silently enable much of today’s AI-based products, with several online platforms offering a myriad of data labelling and content moderation tasks through convenient labour marketplaces. The HCI community has been increasingly interested in investigating the worker-centric issues inherent in the current model and seeking for potential improvements that could be implemented in the future. This workshop explores how a reimagined perspective on crowdsourcing platforms could provide a more equitable, fair, and rewarding experience. This includes not only the workers but also the platforms, who could benefit e.g. from better processes for worker onboarding, skills-development, and growth. We invite visionary takes in various formats on this topic to spread awareness of worker-centric research and developments to the CHI community. As a result of interactive ideation work in the workshop, we articulate a future direction roadmap for research centred around crowdsourcing platforms. Finally, as a specific interest area, the workshop seeks to study crowdwork from the context of the Global South, which has been arising as an important but critically understudied crowdsourcing market in recent years.
@inproceedings{Alorwu2022-oi, title = {{REGROW}: Reimagining Global Crowdsourcing for Better {Human-AI} Collaboration}, booktitle = {Extended Abstracts of the 2022 {CHI} Conference on Human Factors in Computing Systems}, author = {Alorwu, Andy and Savage, Saiph and {van Berkel}, Niels and Ustalov, Dmitry and Drutsa, Alexey and Oppenlaender, Jonas and Bates, Oliver and Hettiachchi, Danula and Gadiraju, Ujwal and Goncalves, Jorge and Hosio, Simo}, publisher = {ACM}, number = {Article 88}, pages = {1--7}, series = {CHI EA '22}, year = {2022}, address = {New York, NY, USA}, location = {New Orleans, LA, USA}, isbn = {9781450391566}, doi = {10.1145/3491101.3503725}, }
2021
- The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can HelpDanula Hettiachchi, Mike Schaekermann, Tristan McKinney, and Matthew LeaseProceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 26, 2021
We consider a class of variable effort human annotation tasks in which the number of labels required per item can greatly vary (e.g., finding all faces in an image, named entities in a text, bird calls in an audio recording, etc.). In such tasks, some items require far more effort than others to annotate. Furthermore, the per-item annotation effort is not known until after each item is annotated since determining the number of labels required is an implicit part of the annotation task itself. On an image bounding-box task with crowdsourced annotators, we show that annotator accuracy and recall consistently drop as effort increases. We hypothesize reasons for this drop and investigate a set of approaches to counteract it. Firstly, we benchmark on this task a set of general best-practice methods for quality crowdsourcing. Notably, only one of these methods actually improves quality: the use of visible gold questions that provide periodic feedback to workers on their accuracy as they work. Given these promising results, we then investigate and evaluate variants of the visible gold approach, yielding further improvement. Final results show a 7% improvement in bounding-box accuracy over the baseline. We discuss the generality of the visible gold approach and promising directions for future research.
@article{Hettiachchi2021-vm, title = {The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help}, author = {Hettiachchi, Danula and Schaekermann, Mike and McKinney, Tristan and Lease, Matthew}, journal = {Proceedings of the ACM on Human-Computer Interaction}, volume = {5}, number = {CSCW2}, pages = {26}, month = may, year = {2021}, doi = {10.1145/3476073}, talk = {https://youtu.be/uN8syzhUnRU}, } - Effect of Conformity on Perceived Trustworthiness of News in Social MediaIEEE Internet Computing, 25(1), 12–19, 2021
A catalyst for the spread of fake news is the existence of comments that users make in support of, or against, such articles. In this article, we investigate whether critical and supportive comments can induce conformity in how readers perceive trustworthiness of news articles and respond to them. We find that individuals tend to conform to the majority’s opinion of an article’s trustworthiness (58%), especially when challenged by larger majorities who are critical of the article’s credibility, or when less confident about their personal judgment. Moreover, we find that individuals who conform are more inclined to take action: to report articles they perceive as fake, and to comment on and share articles they perceive as real. We conclude with a discussion on the implications of our findings for mitigating the dispersion of fake news on social media.
@article{Wijenayake2021-pn, title = {Effect of Conformity on Perceived Trustworthiness of News in Social Media}, author = {Wijenayake, Senuri and Hettiachchi, Danula and Hosio, Simo and Kostakos, Vassilis and Goncalves, Jorge}, journal = {IEEE Internet Computing}, volume = {25}, number = {1}, pages = {12--19}, month = jan, year = {2021}, issn = {1089-7801, 1941-0131}, doi = {10.1109/MIC.2020.3032410}, } - Team Dynamics in Hospital Workflows: An Exploratory Study of a Smartphone Task ManagerDanula Hettiachchi, Lachie Hayes, Jorge Goncalves, and Vassilis KostakosJMIR Medical Informatics, 9(8), e28245, 2021
BACKGROUND: Although convenient and reliable modern messaging apps like WhatsApp enable efficient communication among hospital staff, hospitals are now pivoting toward purpose-built structured communication apps for various reasons, including security and privacy concerns. However, there is limited understanding of how we can examine and improve hospital workflows using the data collected through such apps as an alternative to costly and challenging research methods like ethnography and patient record analysis. OBJECTIVE: We seek to identify whether the structure of the collected communication data provides insights into hospitals’ workflows. Our analysis also aims to identify ways in which task management platforms can be improved and designed to better support clinical workflows. METHODS: We present an exploratory analysis of clinical task records collected over 22 months through a smartphone app that enables structured communication between staff to manage and execute clinical workflows. We collected over 300,000 task records between July 2018 and May 2020 completed by staff members including doctors, nurses, and pharmacists across all wards in an Australian hospital. RESULTS: We show that important insights into how teams function in a clinical setting can be readily drawn from task assignment data. Our analysis indicates that predefined labels such as urgency and task type are important and impact how tasks are accepted and completed. Our results show that both task sent-to-accepted (P<.001) and sent-to-completed (P<.001) times are significantly higher for routine tasks when compared to urgent tasks. We also show how task acceptance varies across teams and roles and that internal tasks are more efficiently managed than external tasks, possibly due to increased trust among team members. For example, task sent-to-accepted time (minutes) is significantly higher (P<.001) for external assignments (mean 22.10, SD 91.45) when compared to internal assignments (mean 19.03, SD 82.66). CONCLUSIONS: Smartphone-based task assignment apps can provide unique insights into team dynamics in clinical settings. These insights can be used to further improve how well these systems support clinical work and staff.
@article{Hettiachchi2021-ql, title = {Team Dynamics in Hospital Workflows: An Exploratory Study of a Smartphone Task Manager}, author = {Hettiachchi, Danula and Hayes, Lachie and Goncalves, Jorge and Kostakos, Vassilis}, journal = {JMIR Medical Informatics}, volume = {9}, number = {8}, pages = {e28245}, month = aug, year = {2021}, language = {en}, issn = {2291-9694}, pmid = {34398797}, doi = {10.2196/28245}, } - Investigating and Mitigating Biases in Crowdsourced DataDanula Hettiachchi, Mark Sanderson, Jorge Goncalves, Simo Hosio, Gabriella Kazai, Matthew Lease, Mike Schaekermann, and Emine YilmazIn Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’21 Companion), 2021
@inproceedings{Hettiachchi2021-gf, title = {Investigating and Mitigating Biases in Crowdsourced Data}, booktitle = {Companion Publication of the 2021 Conference on Computer Supported Cooperative Work and Social Computing}, author = {Hettiachchi, Danula and Sanderson, Mark and Goncalves, Jorge and Hosio, Simo and Kazai, Gabriella and Lease, Matthew and Schaekermann, Mike and Yilmaz, Emine}, publisher = {ACM}, year = {2021}, series = {{CSCW} '21 Companion}, location = {Virtual}, doi = {10.1145/3462204.3481729}, } - Investigating Human Scale Spatial ExperienceProceedings of the ACM on Human-Computer Interaction, 5(ISS), 1–18, 2021
Spatial experience, or how humans experience a given space, has been a pivotal topic especially in urban-scale environments. On the human scale, HCI researchers have mostly investigated personal meanings or aesthetic and embodied experiences. In this paper, we investigate the human scale as an ensemble of individual spatial features. Through large-scale online questionnaires we first collected a rich set of spatial features that people generally use to characterize their surroundings. Second, we conducted a set of field interviews to develop a more nuanced understanding of the feature identified as most important: perceived safety. Our combined quantitative and qualitative analysis contributes to spatial understanding as a form of context information and presents a timely investigation into the perceived safety of human scale spaces. By connecting our results to the broader scientific literature, we contribute to the field of HCI spatial understanding.
@article{Paananen2021-mz, title = {Investigating Human Scale Spatial Experience}, author = {Paananen, Ville and Oppenlaender, Jonas and Goncalves, Jorge and Hettiachchi, Danula and Hosio, Simo}, journal = {Proceedings of the ACM on Human-Computer Interaction}, publisher = {ACM}, volume = {5}, number = {ISS}, pages = {1--18}, year = {2021}, address = {New York, NY, USA}, keywords = {architecture, spatial experience, interaction design}, doi = {10.1145/3488541}, }
2020
- “Hi! I am the Crowd Tasker” Crowdsourcing through Digital Voice AssistantsDanula Hettiachchi, Zhanna Sarsenbayeva, Fraser Allison, Niels van Berkel, Tilman Dingler, Gabriele Marini, Vassilis Kostakos, and Jorge GoncalvesIn Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 2020
Inspired by the increasing prevalence of digital voice assistants, we demonstrate the feasibility of using voice interfaces to deploy and complete crowd tasks. We have developed Crowd Tasker, a novel system that delivers crowd tasks through a digital voice assistant. In a lab study, we validate our proof-of-concept and show that crowd task performance through a voice assistant is comparable to that of a web interface for voice-compatible and voice-based crowd tasks for native English speakers. We also report on a field study where participants used our system in their homes. We find that crowdsourcing through voice can provide greater flexibility to crowd workers by allowing them to work in brief sessions, enabling multi-tasking, and reducing the time and effort required to initiate tasks. We conclude by proposing a set of design guidelines for the creation of crowd tasks for voice and the development of future voice-based crowdsourcing systems.
@inproceedings{Hettiachchi2020-qj, title = {``Hi! {I} am the Crowd Tasker'' Crowdsourcing through Digital Voice Assistants}, booktitle = {Proceedings of the 2020 {CHI} Conference on Human Factors in Computing Systems}, author = {Hettiachchi, Danula and Sarsenbayeva, Zhanna and Allison, Fraser and {van Berkel}, Niels and Dingler, Tilman and Marini, Gabriele and Kostakos, Vassilis and Goncalves, Jorge}, publisher = {ACM}, pages = {1--14}, series = {CHI '20}, month = apr, year = {2020}, address = {New York, NY, USA}, isbn = {9781450367080}, doi = {10.1145/3313831.3376320}, } - CrowdCog: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in CrowdsourcingProceedings of the ACM on Human-Computer Interaction, 4(CSCW2), 1–22, 2020
While crowd workers typically complete a variety of tasks in crowdsourcing platforms, there is no widely accepted method to successfully match workers to different types of tasks. Researchers have considered using worker demographics, behavioural traces, and prior task completion records to optimise task assignment. However, optimum task assignment remains a challenging research problem due to limitations of proposed approaches, which in turn can have a significant impact on the future of crowdsourcing. We present ’CrowdCog’, an online dynamic system that performs both task assignment and task recommendations, by relying on fast-paced online cognitive tests to estimate worker performance across a variety of tasks. Our work extends prior work that highlights the effect of workers’ cognitive ability on crowdsourcing task performance. Our study, deployed on Amazon Mechanical Turk, involved 574 workers and 983 HITs that span across four typical crowd tasks (Classification, Counting, Transcription, and Sentiment Analysis). Our results show that both our assignment method and recommendation method result in a significant performance increase (5% to 20%) as compared to a generic or random task assignment. Our findings pave the way for the use of quick cognitive tests to provide robust recommendations and assignments to crowd workers.
@article{Hettiachchi2020-bc, title = {{CrowdCog}: A Cognitive Skill based System for Heterogeneous Task Assignment and Recommendation in Crowdsourcing}, author = {Hettiachchi, Danula and {van Berkel}, Niels and Kostakos, Vassilis and Goncalves, Jorge}, journal = {Proceedings of the ACM on Human-Computer Interaction}, publisher = {ACM}, volume = {4}, number = {CSCW2}, pages = {1--22}, month = oct, year = {2020}, address = {New York, NY, USA}, doi = {10.1145/3415181}, talk = {https://youtu.be/5c6ZXDGqrqk}, } - How Context Influences Cross-Device Task Acceptance in Crowd WorkIn Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (HCOMP ’20), 2020
Although crowd work is typically completed through desktop or laptop computers by workers at their home, literature has shown that crowdsourcing is feasible through a wide array of computing devices, including smartphones and digital voice assistants. An integrated crowdsourcing platform that operates across multiple devices could provide greater flexibility to workers, but there is little understanding of crowd workers’ perceptions on uptaking crowd tasks across multiple contexts through such devices. Using a crowdsourcing survey task, we investigate workers’ willingness to accept different types of crowd tasks presented on three device types in different scenarios of varying location, time and social context. Through analysis of over 25,000 responses received from 329 crowd workers on Amazon Mechanical Turk, we show that when tasks are presented on different devices, the task acceptance rate is 80.5% on personal computers, 77.3% on smartphones and 70.7% on digital voice assistants. Our results also show how different contextual factors such as location, social context and time influence workers decision to accept a task on a given device. Our findings provide important insights towards the development of effective task assignment mechanisms for cross-device crowd platforms.
@inproceedings{Hettiachchi2020-ei, title = {How Context Influences {Cross-Device} Task Acceptance in Crowd Work}, booktitle = {Proceedings of the {AAAI} Conference on Human Computation and Crowdsourcing}, author = {Hettiachchi, Danula and Wijenayake, Senuri and Hosio, Simo and Kostakos, Vassilis and Goncalves, Jorge}, publisher = {AAAI}, volume = {8}, pages = {53--62}, year = {2020}, series = {HCOMP '20}, doi = {https://doi.org/10.1609/hcomp.v8i1.7463}, talk = {https://youtu.be/MwzZZupcSg0}, }
2019
- Effect of Cognitive Abilities on Crowdsourcing Task PerformanceIn Human-Computer Interaction – INTERACT 2019, 2019
Matching crowd workers to suitable tasks is highly desirable as it can enhance task performance, reduce the cost for requesters, and increase worker satisfaction. In this paper, we propose a method that considers workers’ cognitive ability to predict their suitability for a wide range of crowdsourcing tasks. We measure cognitive ability via fast-paced online cognitive tests with a combined average duration of 6.2 min. We then demonstrate that our proposed method can effectively assign or recommend workers to five different popular crowd tasks: Classification, Counting, Proofreading, Sentiment Analysis, and Transcription. Using our approach we demonstrate a significant improvement in the expected overall task accuracy. While previous methods require access to worker history or demographics, our work offers a quick and accurate way to determine which workers are more suitable for which tasks.
@inproceedings{Hettiachchi2019-lo, title = {Effect of Cognitive Abilities on Crowdsourcing Task Performance}, booktitle = {{Human-Computer} Interaction -- {INTERACT} 2019}, author = {Hettiachchi, Danula and {van Berkel}, Niels and Hosio, Simo and Kostakos, Vassilis and Goncalves, Jorge}, publisher = {Springer}, pages = {442--464}, year = {2019}, doi = {10.1007/978-3-030-29381-9\_28}, } - Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case StudyNiels van Berkel, Jorge Goncalves, Danula Hettiachchi, Senuri Wijenayake, Ryan M Kelly, and Vassilis KostakosProceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–21, 2019
The increased reliance on algorithmic decision-making in socially impactful processes has intensified the calls for algorithms that are unbiased and procedurally fair. Identifying fair predictors is an essential step in the construction of equitable algorithms, but the lack of ground-truth in fair predictor selection makes this a challenging task. In our study, we recruit 90 crowdworkers to judge the inclusion of various predictors for recidivism. We divide participants across three conditions with varying group composition. Our results show that participants were able to make informed decisions on predictor selection. We find that agreement with the majority vote is higher when participants are part of a more diverse group. The presented workflow, which provides a scalable and practical approach to reach a diverse audience, allows researchers to capture participants’ perceptions of fairness in private while simultaneously allowing for structured participant discussion.
@article{Van_Berkel2019-aw, title = {Crowdsourcing Perceptions of Fair Predictors for Machine Learning: A Recidivism Case Study}, author = {{van Berkel}, Niels and Goncalves, Jorge and Hettiachchi, Danula and Wijenayake, Senuri and Kelly, Ryan M and Kostakos, Vassilis}, journal = {Proceedings of the ACM on Human-Computer Interaction}, publisher = {ACM}, volume = {3}, number = {CSCW}, pages = {1--21}, year = {2019}, address = {New York, NY, USA}, doi = {10.1145/3359130}, } - Measuring the Effects of Stress on Mobile InteractionZhanna Sarsenbayeva, Niels van Berkel, Danula Hettiachchi, Weiwei Jiang, Tilman Dingler, Eduardo Velloso, Vassilis Kostakos, and Jorge GoncalvesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(1), 1–18, 2019
Research shows that environmental factors such as ambient noise and cold ambience can render users situationally impaired, adversely affecting interaction with mobile devices. However, an internal factor which is known to negatively impact cognitive abilities – stress – has not been systematically investigated in terms of its impact on mobile interaction. In this paper, we report a study where we use the Trier Social Stress Test to induce stress on participants, and investigate its effect on three aspects of mobile interaction: target acquisition, visual search, and text entry. We find that stress reduces completion time and accuracy during target acquisition tasks, as well as completion time during visual search tasks. Finally, we are able to directly contrast the magnitude of these effects to previously published effects of environmentally-caused impairments. Our work contributes to the growing body of literature on situational impairments.
@article{Sarsenbayeva2019-mk, title = {Measuring the Effects of Stress on Mobile Interaction}, author = {Sarsenbayeva, Zhanna and {van Berkel}, Niels and Hettiachchi, Danula and Jiang, Weiwei and Dingler, Tilman and Velloso, Eduardo and Kostakos, Vassilis and Goncalves, Jorge}, journal = {Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}, publisher = {ACM}, volume = {3}, number = {1}, pages = {1--18}, year = {2019}, address = {New York, NY, USA}, doi = {10.1145/3314411}, } - Towards Effective Crowd-Powered Online Content ModerationDanula Hettiachchi, and Jorge GoncalvesIn Proceedings of the 31st Australian Conference on Human-Computer-Interaction (OZCHI ’19), 2019
Content moderation is an important element of social computing systems that facilitates positive social interaction in online platforms. Current solutions for moderation including human moderation via commercial teams are not effective and have failed to meet the demands of growing volumes of online user generated content. Through a study where we ask crowd workers to moderate tweets, we demonstrate that crowdsourcing is a promising solution for content moderation. We also report a strong relationship between the sentiment of a tweet and its appropriateness to appear in public media. Our analysis on worker responses further reveals several key factors that affect the judgement of crowd moderators when deciding on the suitability of text content. Our findings contribute towards the development of future robust moderation systems that utilise crowdsourcing.
@inproceedings{Hettiachchi2019-fi, title = {Towards Effective {Crowd-Powered} Online Content Moderation}, booktitle = {Proceedings of the 31st Australian Conference on {Human-Computer-Interaction}}, author = {Hettiachchi, Danula and Goncalves, Jorge}, publisher = {ACM}, pages = {342--346}, series = {OZCHI '19}, year = {2019}, address = {New York, NY, USA}, location = {Fremantle, WA, Australia}, isbn = {9781450376969}, doi = {10.1145/3369457.3369491}, } - Effect of Ambient Light on Mobile InteractionZhanna Sarsenbayeva, Niels van Berkel, Weiwei Jiang, Danula Hettiachchi, Vassilis Kostakos, and Jorge GoncalvesIn Human-Computer Interaction (INTERACT ’19), 2019
@inproceedings{Sarsenbayeva2019-cn, title = {Effect of Ambient Light on Mobile Interaction}, author = {Sarsenbayeva, Zhanna and {van Berkel}, Niels and Jiang, Weiwei and Hettiachchi, Danula and Kostakos, Vassilis and Goncalves, Jorge}, booktitle = {Human-Computer Interaction}, pages = {465--475}, year = {2019}, series = {INTERACT '19}, doi = {10.1007/978-3-030-29387-1\_26}, } - Enabling Creative Crowd Work Through Smart SpeakersDanula Hettiachchi, Niels van Berkel, Tilman Dingler, Fraser Allison, Vassilis Kostakos, and Jorge GoncalvesIn Workshop on Designing Crowd-powered Creativity Support Systems (CHI’19 Workshop), 2019
@inproceedings{Hettiachchi2019-pw, title = {Enabling Creative Crowd Work Through Smart Speakers}, booktitle = {Workshop on Designing Crowd-powered Creativity Support Systems}, author = {Hettiachchi, Danula and {van Berkel}, Niels and Dingler, Tilman and Allison, Fraser and Kostakos, Vassilis and Goncalves, Jorge}, pages = {1--5}, series = {{CHI'19} Workshop}, year = {2019}, doi = {https://doi.org/10.5281/zenodo.2648986}, }