Computer Science Distinguished Speaker: Ani Kembhavi

Description

Aniruddha Kembhavi, the senior director of computer vision at the Allen Institute for AI and an associate professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington, will give a talk titled "Building Foundation Models for Vision and Robotics" for the Department of Computer Science. This lecture was rescheduled from November 2023.

Abstract:

Large language models support a whole gamut of tasks in natural language. In contrast, unification has been more challenging in computer vision, partly due to the heterogeneity of tasks in the visual domain and a scarcity of robotics data in the physical world. How do we create powerful unified systems for vision and robotics that can be as capable and creative as their language counterparts? Aniruddha Kembhavi will first present his work on Unified IO, the first single neural model to perform a large and diverse set of AI tasks spanning classical computer vision, image synthesis, vision-and-language, and natural language processing, and his follow-up work, Unified IO 2, that brings in video, audio, and action. Then he will present his recent surprising finding in robotics: Imitating shortest path planners in simulation can produce agents that can proficiently navigate, explore, and manipulate objects in the real world, with no human data or RL and with just RGB sensing—made possible by his recent works on ProcTHOR, HoloDeck, and ObjaVerse. Finally, he will present a compelling alternative to building unified models, Visual Programming, which uses language models as code generators to invoke smaller specialized vision models. This paradigm is efficient and effective, leverages models sourced from the entire community, and scales easily to large sets of diverse tasks.