LCSR Seminar: Sherry Yang

Description

Sherry Yang, a final-year doctoral student at UC Berkeley and a senior research scientist at Google DeepMind, will give a talk titled "Decision-Making with Internet-Scale Knowledge" for the Laboratory for Computational Sensing + Robotics.

Abstract:

Machine learning models pretrained on internet data have acquired broad knowledge about the world but struggle to solve complex tasks that require extended reasoning and planning. Sequential decision-making, on the other hand, has empowered AlphaGo's superhuman performance, but lacks visual, language, and physical knowledge about the world. In this talk, I will present my research toward enabling decision-making with internet-scale knowledge. First, I will illustrate how language models and video generation are unified interfaces that can integrate internet knowledge and represent diverse tasks, enabling the creation of a generative simulator to support real-world decision-making. Second, I will discuss my work on designing decision-making algorithms that can take advantage of generative language and video models as agents and environments. Combining pretrained models with decision-making algorithms can effectively enable a wide range of applications such as developing chatbots, learning robot policies, and discovering novel materials.

Sherry Yang's research aims to develop machine learning models with internet-scale knowledge to make better-than-human decisions. To this end, she has developed techniques for generative modeling and representation learning from large-scale vision, language, and structured data, coupled with developing algorithms for sequential decision-making such as imitation learning, planning, and reinforcement learning. Yang initiated and led the Foundation Models for Decision-Making workshop at NeurIPS (the Conference on Neural Information Processing Systems) 2022 and 2023, bringing together research communities in vision, language, planning, and reinforcement learning to solve complex decision-making tasks at scale. Before her current role, Yang received her bachelor's and master's degrees from MIT, advised by Patrick Winston and Julian Shun. Her doctoral adviser is Pieter Abbeel.