CS/CLSP Spring 2025 Seminar Series: Ting Hao Huang

Description

Ting-Hao "Kenneth" Huang, an associate professor at the College of Information Sciences and Technology at Penn State, will give a talk titled "What Roles Will Humans Play in the Future of Data Annotation" for the Department of Computer Science and the Center for Language and Speech Processing.

Abstract

Large language models (LLMs) have rapidly and impressively taken over many tasks once handled by human annotators in constructing text datasets. But does this mean we no longer need humans in the annotation loop? What roles should humans play in future data annotation pipelines? In this talk, I will present two recent studies that explore the evolving role of humans in the landscape of text data annotation. First, we ask whether a well-designed and carefully executed traditional crowdsourcing pipeline can still outperform LLMs in labeling quality. Our study offers an in-depth and holistic comparison between human and LLM annotation performance. Second, we turn to a future where LLMs increasingly replace manual annotation labor. In this scenario, the human role shifts toward instructing the models–often through prompting. But how effective are humans at prompting LLMs for annotation tasks, especially when working without access to gold-standard labels? We investigate this growing practice, which we call "prompting in the dark," and assess its implications for the quality and reliability of LLM-generated annotations.