Robots are making progress at autonomously generating actions that optimize specified objective functions. In my lab and together with our collaborators, we work on how to do this optimization when the robot is not acting in isolation, but is sharing the environment with people, and its actions need to end up coordinating well with theirs. More recently, we’ve also turned our attention to what the objective function should be in the first place. We begun to challenge the notion of a perfect pre-specified objective, and instead introduced algorithms that help robots optimize for what people internally want, even when they can’t robustly specify it. We apply these algorithms across different domains, like autonomous driving and collaborative manipulation.