Human-Object Interaction Retargeting

Retargeting human-object interactions to robots using constrained optimisation and reinforcement learning.

Overview

As part of my semester project at RSL (Robotic Systems Lab) for my Master’s degree, I am working on retargeting human-object interactions to robots using constrained optimisation and reinforcement learning. The goal of this project is to create physically plausible retargeted data of human demonstrations that can be used for downstream environments. The project is still ongoing, and is expected to be completed by the end of June 2026.

Retargeting human-object interaction data to robots.

Demo

We are given a source human demonstration in SMPL-X format from the OMOMO dataset:

Start with an inverse kinematics retargeter, which is limited by human-robot embodiment gap:

Then, constrained optimisation is used to encourage correct contacts points:

And finally, reinforcement learning is used to create a physically plausible result: