Fall 2025 GRASP SFI: Minghan Zhu, University of Pennsylvania, “3D Robot Vision for Structured World Understanding”

This presenter is one of the winners of the 2025 GRASP vote for internal PhD or postdoc SFI Speakers!

This was a hybrid event with in-person attendance in Levine 307 and virtual attendance…

ABSTRACT

Deploying robots in diverse real-world environments is a fundamental challenge. While recent AI advances are impressive, robots still struggle to generalize. I argue that a key missing piece in embodied intelligence is the structured understanding of the world: how geometries compose, how physics governs interactions, and how dynamics unfold. My research in 3D vision develops this understanding with two complementary principles: physics-based reasoning and symmetry-aware learning. First, I present Vysics, fusing vision and contact-rich physics to overcome heavy occlusions in object reconstruction, and my recent follow-up work that further incorporates 3D generative priors for reconstructions with both high visual fidelity and physical compliance. Then, I demonstrate my work on leveraging symmetry for efficient modeling of 3D geometry and dynamics. I introduce my algorithmic contributions in equivariant learning, including E2PN, which improves the efficiency of point cloud learning by 7x compared with prior work, and Lie Neurons and Reductive Lie Neurons, which expand the scope of symmetry preserved by equivariant networks from rotations to general linear transformations. These advances enable significant progress in various robotic tasks by incorporating symmetry, from segmentation and place recognition to odometry and dynamics learning. I will close with my vision of building structured world representations that are simultaneously grounded in physics, informed by data, and structured by symmetry, toward robots that truly understand their physical world.