Loading Events

« All Events

  • This event has passed.

Fall 2023 GRASP SFI: Jim Fan, NVIDIA AI, “Generalist Agents in Open-Ended Worlds”

September 13 @ 3:00 pm - 4:00 pm

This is a hybrid event with in-person attendance in Levine 307 and virtual attendance on Zoom.


Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports an infinite variety of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with 1000s of diverse open-ended tasks and an internet-scale knowledge base with YouTube videos, Wiki pages, and Reddit posts. We also propose two new algorithms on top of MineDojo: 1) MineCLIP, a foundation reward function reminiscent of RLHF for embodied agents; and 2) Voyager, an LLM-powered lifelong learning agent that explores and improves itself purely in-context. We look forward to seeing how MineDojo empowers the community to make more progress on the grand challenge of open-ended agent learning.


Jim Fan

Jim Fan - Learn More

Jim Fan is a senior AI research scientist at NVIDIA. His mission is to build generally capable AI agents with applications to gaming, robotics, and software automation. His research spans foundation models, multimodal AI, reinforcement learning, and open-ended learning. Jim obtained his Ph.D. degree in Computer Science from Stanford University, advised by Prof. Fei-Fei Li. His work “MineDojo” won the Outstanding Paper Award at NeurIPS 2022. Previously, Jim did research internships at OpenAI, Google AI, and MILA-Quebec AI Institute. He was the Valedictorian of the class of 2016 and a recipient of the Illig Medal at Columbia University. Personal website: https://jimfan.me


September 13
3:00 pm - 4:00 pm
Event Category:


Levine 307
3330 Walnut St
Philadelphia, PA 19104 United States
+ Google Map