Spring 2026 GRASP on Robotics: Roni Sengupta, University of North Carolina at Chapel Hill, “From Pixels to Physics: Understanding and Manipulating Physics from Images”

This event will be in-person ONLY in Wu and Chen Auditorium.

ABSTRACT

A hallmark of human vision is the ability to reason about the physics of the world: we can infer the shape of the object, how light reflects off the object, and how the object deforms under force. Yet today’s AI systems still lack this kind of physical intuition. Enabling machines to perceive and manipulate physics would mark a major step toward grounding AI in the real world.

In this talk, I will present my lab’s research at the intersection of computer vision, graphics, and machine learning that takes an inverse graphics perspective: rather than forward-simulating the physics of light, materials, and deformation as in decades of computer graphics, we aim to invert this process to infer and control these properties directly from images. The first part of the talk will focus on explicitly estimating physics, with examples in recovering lighting, reflectance, and object deformation. The second part will show how controlled generative models allow us to manipulate physics, with applications in relighting and simulating facial aging. Together, these efforts highlight a path toward AI systems with a deeper and more actionable understanding of the physical world.