Recording human tactile data as training data for robots

Glove-type and exoskeleton-type devices are widely used to collect human demonstration data for imitation and reinforcement learning. They capture joint angles of the hand and fingertip contact or pressure with high accuracy — but the form itself has structural limits.

• Covering the hand changes how the bare hand moves

  Gloves and exoskeletons that cover the palm and fingers affect both contact sensation and range of motion during grasping and manipulation. Research has pointed out both that closed-palm data gloves restrict natural hand movement, and that the palm plays a large part in stable grasping.

• Capturing force means covering the hand further

  Adding tactile or pressure sensors to the fingertips or palm to obtain force information extends how much of the hand is covered. Keeping movement close to that of the bare hand and capturing force information end up in tension with each other, on the hand itself.

The movement data collected while wearing such a device therefore tends to diverge from natural bare-handed movement, and that divergence is carried straight into the training data.

Our data wearables address this trade-off by not concentrating contact and force on the hand, capturing each with the minimum amount worn.

• Fingertip vibration sensors

  Vibration at the surface of the fingertip skin reveals the onset of contact and changes in the texture of the contacting surface. Only part of the fingertip is covered, so the palm and the range of finger motion stay free.

• FMG (force myography) on the forearm

  Nothing is worn on the hand: the magnitude of force is captured from the deformation of the forearm muscles.

Combining the timing of contact from the vibration sensors with the magnitude of force from FMG along a shared timeline yields a dataset that ties how much force was applied to which contact. Because the palm stays uncovered, this can be collected while movement stays close to that of the bare hand.