Fall 2026 GRASP on Robotics: James Anderson, Columbia University, “How to Control Your JEPA”

This will be an in-person event ONLY in Wu and Chen Auditorium.

ABSTRACT

Planning and control based on visual observations require representations that preserve what matters for decision making, not merely what is predictable. Visual observations contain many features that may be easy to predict yet irrelevant to the decisions a controller must make. The central representation-learning problem is therefore not just to construct an accurate latent predictor, but to construct a latent state that facilitates control design, i.e., keeps what is beneficial to the planning problem while discarding what distracts from it. Joint-embedding predictive architectures (JEPAs) provide an appealing framework for learning such latent dynamics without reconstructing observations, but standard predictive objectives do not by themselves guarantee this control-relevant structure. I will describe our recent work using bisimulation to impose control-relevant state abstraction, encouraging observations with similar dynamics and outcomes to map to nearby latent states. This improves robustness to visual distractors and slow features while producing latent spaces up to 10× smaller than those used by DINO-WM.

More broadly, I will ask how latent spaces should be designed when their ultimate purpose is planning and control rather than prediction alone. Bisimulation provides one mechanism for determining what a representation should ignore; complementary ideas such as invariance, equivariance, and task-aware latent geometry can shape what structure it should preserve. The goal is to develop world models whose representations are explicitly designed to support efficient optimization, planning, and closed-loop feedback.