Spring 2026 GRASP SFI: Junyao Shi, University of Pennsylvania, “Unlocking Generalist Robots with Human Data and Foundation Models”

Building general-purpose robots remains fundamentally constrained by data scarcity and labor-intensive engineering. Unlike vision and language, robotics lacks large, diverse datasets spanning tasks, environments, and embodiments, limiting both scalability and generalization.

This talk explores how human data and foundation models trained at scale can help overcome these bottlenecks. I will discuss recent progress in leveraging in-the-wild human videos to transfer manipulation skills and provide rich signals for robot learning. I will also highlight how vision and language models can automate traditionally hand-engineered components of robotics pipelines, including modular system construction, data collection, reward design, and simulation construction.

Together, these directions point toward a new paradigm for robot learning: one that replaces task-specific engineering with scalable learning from non-robot knowledge and data, bringing us closer to adaptable, generalist robots.