GRASP Lab AI in the Physical World: Helping Robotic Arms Solve Problems Using Tools

When Tianyu Li, a doctoral student in Mechanical Engineering and Applied Mechanics (MEAM), and George Jiayuan Gao, former master’s student in Robotics and current research engineer at Dyna, set out to study how robots might better interact with the physical world, they weren’t just thinking about smarter machines, they were thinking about tools.

“Humans don’t just rely on their bodies to solve problems,” Li explains. “We design tools to use.”

That simple idea sits at the core of a new research project from Li and Gao with their advisors, Nadia Figueroa, Shalini and Rajeev Misra Presidential Assistant Professor in MEAM, and Dinesh Jayaraman, Assistant Professor in Computer and Information Science. The full research team also includes Junyao Shi, doctoral student in CIS, Yihan Li and Zizhe Zhang, both master’s students in Robotics through the ROBO Program, jointly sponsored by CIS, Electrical Systems Engineering (ESE) and MEAM. Presented by Li at the International Conference on Learning Representations (ICLR) 2026 in Rio de Janeiro, the team’s project, VLMgineer, is a framework that uses artificial intelligence not just to “think,” but to design, adapt and deploy tools in the real world.

_{Tianyu Li presenting a poster on VLMgineer at ICLR.}

From Chatbots to Physical Intelligence

Today’s large AI models are often associated with chatbots, systems that generate text, answer questions or write code. But Li’s research pushes beyond conversation into something more tangible: physical reasoning.

“What we’re asking,” he says, “is how large AI models can understand space, objects and actions and then help a robot actually do something.”

With VLMgineer, AI observes a task through visual input like a robot attempting to pick up scattered objects on a table. Instead of directly controlling the robot or rewriting its programming, the system takes a different approach: it suggests tool designs.

The AI proposes designs, simulates their effectiveness, iterates on improvements and ultimately generates a refined solution. In some cases, those designs can be fabricated, using technologies like 3D printing, and tested in the real world. The system can then observe the outcome and continue learning.

It’s a cycle that mirrors how humans and animals learn: observe, experiment, adapt.

“This project is a reflection of my philosophy on how AI models should be used, not as all-knowing oracles, but as statistically plausible generators,” says Figueroa. “I see generative models as imperfect tools but they can be very powerful if combined with structured, verifiable approaches. This is exactly what we are doing with VLMgineer.”

“What I find exciting about this project is that tool design and use have long been thought of as key markers of intelligence in natural organisms, like crows and chimps,” adds Jayaraman. “This project shows a route through which today’s foundation models can imbue robots with those abilities.”

Letting AI Take the Lead

A key part of the project is restraint.

“We avoid incorporating any human feedback when generating suggestions for each task,” says Li. “We want to test the AI’s ability to problem-solve without human intervention.”

This autonomy is critical for scalability. If AI systems can independently design and refine tools, they could dramatically reduce the need for manual engineering in robotics, especially in complex or unpredictable environments.

But sometimes, the ideas are unconventional or even physically unrealistic.

“That’s one of the challenges,” says Li. “The AI doesn’t always understand real-world constraints like materials or manufacturing limits yet.”

Bridging that gap is a major focus of ongoing work.

A New Way To Think About Robotics

Traditionally, robotics research has focused on improving the robot itself: its hardware, control systems or precision. The team’s work suggests a shift in perspective: instead of redesigning the robot, equip it with better tools.

It’s an approach inspired by everyday life.

“Think about cooking,” Li says. “We use all kinds of tools to accomplish very specific tasks. Spatulas, garlic presses and pot holders all have their unique purposes that allow us to adapt to the problem at hand with a simple solution. We don’t try to change our hands to do those tasks.”

This philosophy could reshape how robots are deployed across industries. In manufacturing, robots might design custom tools to assemble complex parts. In laboratories, they could develop specialized instruments for handling delicate materials. In agriculture, they might adapt tools for tasks like watering or harvesting.

Even household robots could benefit. Imagine a vacuum that not only navigates a room, but recognizes obstacles, picks up clutter and chooses the right tool for the job.

“Ever since the rise of coding models, my group has been building tools to exploit them for training sophisticated robot controllers in simulation for tasks like getting quadruped robots to navigate through obstacle courses,” says Jayaraman. “In this project, Tianyu and Jiayuan have explored an exciting alternative: if you instead re-design the robot’s physical interface with the world, you can often get away with quite simple controllers. This is embodied intelligence: shifting the sophistication from the software to the hardware.”

Collaboration and Looking Ahead

The project reflects the interdisciplinary environment at Penn Engineering, where Li credits collaboration across fields, including robotics, machine learning and vision-language modeling, as essential to its success.

“These areas don’t always overlap naturally,” he says. “But when they do, you start to see entirely new possibilities.”

“Robotics requires people from across traditional disciplinary boundaries to work together to make real progress,” adds Jayaraman. “The GRASP lab at Penn Engineering provides a great environment for that to happen.”

As the technology evolves, Li envisions systems that go even further, yielding AI that doesn’t just design tools, but entire robotic ecosystems: the robot, the tool and the strategy, all generated from a single real-world problem.

In the nearer term, the focus is on refining realism and expanding applications. Integrating physical constraints, improving manufacturability and enabling systems to combine existing tools are all active areas of development.

The presentation of this work at ICLR marks an important milestone for both the research and for Li himself, who will graduate this year and is looking to bring this work into industry.

“This kind of system could transform how we approach efficiency and problem-solving,” says Li. “But it will take collaboration to bring it to real-world scale.”

Learn more about the research being conducted at the Figueroa Robotics Lab here and the research coming out of Jayaraman’s Perception, Action, and Learning (PennPAL) Research Group here.