CS Seminar: Mohit Iyyer

Description

Mohit Iyyer, an associate professor in computer science at the University of Massachusetts Amherst, will give a talk titled "Improving, Evaluating, and Detecting Long-Form LLM-Generated Text" for the Department of Computer Science.

Abstract:

Recent advances in large language models (LLMs) have enabled them to process texts exceeding 100,000 tokens in length, fueling demand for long-form language processing tasks such as the summarization or translation of books. However, LLMs struggle to take full advantage of the information within such long contexts, which contributes to factually incorrect and incoherent text generation. In this talk, Mohit Iyyer will first demonstrate an issue that plagues even modern LLMs: their tendency to assign high probability to implausible long-form continuations of their input. He will then describe a contrastive sequence-level ranking model that mitigates this problem at decoding time and that can also be adapted to the reinforcement learning from human feedback alignment paradigm. Next, he will consider the growing problem of long-form evaluation: As the length of the inputs and outputs of long-form tasks grows, how do we even measure progress (via both humans and machines)? He proposes a high-level framework that first decomposes a long-form text into simpler atomic units before then evaluating each unit on a specific aspect. He demonstrates the framework's effectiveness at evaluating factuality and coherence on tasks such as biography generation and book summarization. He will also discuss the rapid proliferation of LLM-generated long-form text, which plagues not only evaluation (e.g., via Mechanical Turkers using ChatGPT to complete tasks) but also society as a whole, and he will describe novel watermarking strategies to detect such text. Finally, he will conclude by discussing his future research vision, which aims to extend long-form language processing to multilingual, multimodal, and collaborative human-centered settings.