Fall 2013 GRASP Seminar: James Rehg, Georgia Institute Of Technology, "Egocentric Recognition Of Objects And Activities"

Presenter: James Rehg (Homepage)

Event Dates:
  Friday November 15, 2013 from 11:00am to 12:00pm

Advances in camera miniaturization and mobile computing have enabled the development of wearable camera systems which can capture both the user's view of the scene (the egocentric, or first-person, view) and their gaze behavior. In contrast to the established third-person video paradigm, the egocentric paradigm makes it possible to easily collect examples of naturally-occurring human behavior, such as activities of daily living, from a consistent vantage point. Moreover, there exist a variety of egocentric cues which can be extracted from these videos and used for weakly-supervised learning of objects and activities. We focus on activities requiring hand-eye coordination and model the spatio-temporal relationship between the gaze point, the scene objects, and the action label. We demonstrate that gaze measurement can provide a powerful cue for recognition. In addition, we present an inference method that can predict gaze locations and use the predicted gaze to infer action labels. We demonstrate improvements in action recognition rates and gaze prediction accuracy relative to state-of-the-art methods, on a new dataset containing egocentric videos of daily activities and gaze. We will also describe some applications in psychology, where we are developing methods for automating the measurement of children's behavior, as part of a large effort targeting autism and other behavioral disorders. This is joint work with Alireza Fathi, Yin Li, and Agata Rozga.

Presenter's Biography:

James M. Rehg (pronounced "ray") is a Professor in the School of Interactive Computing at the Georgia Institute of Technology, where he is co-Director of the Computational Perception Lab ( and is the Associate Director for Research in the Center for Robotics and Intelligent Machines ( (RIM@GT). He received his Ph.D. from CMU in 1995 and worked at the Cambridge Research Lab of DEC (and then Compaq) from 1995-2001, where he managed the computer vision research group. He received an NSF CAREER award in 2001 and a Raytheon Faculty Fellowship from Georgia Tech in 2005. He and his students have received a number of best paper awards, including best student paper awards at ICML 2005 and BMVC 2010. Dr. Rehg serves on the Editorial Board of the Intl. J. of Computer Vision, and he served as the General co-Chair for CVPR 2009. He has authored more than 100 peer-reviewed scientific papers and holds 23 issued US patents. His research interests include computer vision, medical imaging, robot perception, machine learning, and pattern recognition. Dr. Rehg is currently leading a multi-institution effort to develop the science and technology of Behavior Imaging— the capture and analysis of social and communicative behavior using multi-modal sensing, to support the study and treatment of developmental disorders such as autism. See for details.