Perception systems form a crucial part of autonomous and artificial intelligence systems since they convert data about the relationship between an autonomous system and its environment into meaningful information. This information is then processed to produce optimal actions towards achieving desired goals. Perception systems can be difficult to build since they may involve modeling complex physical systems or other autonomous agents. In such scenarios, data driven models may be used to augment physics based models for perception. In this talk, I will present work making use of data driven models for perception tasks, highlighting the benefit of such approaches for autonomous systems. I will present work with application to pedestrian perception for autonomous vehicles and 3D shape and pose estimation for indoor scene understanding and reconstruction.