Teleoperation that conveys force, not just contact

For: Robotics companies and operators on site

The problem

Teleoperation using the controllers of consumer VR devices such as Meta Quest is now widely adopted for operating robots. In most cases, though, the haptic feedback those VR controllers provide is limited to vibration. Recent controllers can control amplitude and frequency finely, but this way of presenting touch has structural limits.

  • It carries no directional information, in principle

    Vibration can express intensity through amplitude and frequency, but it has no means of conveying which way a force is acting. The inability to present force as a vector — direction plus magnitude — is a limit of vibration as a modality, not of any particular device.

  • Sustained changes in force are hard to convey stably

    Presenting vibration continuously leads to perceptual adaptation: the same intensity is gradually felt less. That makes it well suited to signalling momentary contact events, but poorly suited to tracking the magnitude of a sustained force accurately.

  • Absolute intensity discrimination is coarse

    Asking someone to judge force in absolute terms from amplitude alone leaves only a limited number of levels they can reliably distinguish, which caps how finely they can judge the force being applied.

    Prior work reports on the order of three to four levels. To be verified: this depends on the device and the conditions.

As a result, operators are left leaning heavily on visual information to infer how much force they are applying — applying too much and damaging the object, or too little and failing to grip or manipulate it.

Our approach

commissure addresses this with multi-modal feedback through force and tactile display.

  • Fingertip vibration

    Conveys the onset of contact clearly, so that the moment of touching an object or a change in texture can be perceived. It is used for what vibration is genuinely good at: signalling an instantaneous event.

  • Skin stretch (rotational shear)

    Used to present force with direction: the direction and magnitude of the torque or reaction force estimated on the robot side is mapped to rotational shear on the skin. It complements what vibration cannot do — direction, and sustained changes in force.

With vibration carrying discrete contact events and skin stretch carrying the continuous force vector, the operator can perceive the moment of contact separately from how the force changes and where it acts afterwards.

What we expect

A better operating experience

Perceiving the direction of force directly removes the need to infer it from visual information alone, easing cognitive load during operation.

Better operating precision

In tasks where force control is critical — insertion, or gripping fragile objects — more precise operation should become possible.

Carrying over into imitation learning data

A teleoperation log recorded down to the tactile channel is, as it stands, teaching data for robot imitation learning. The more operating a site does, the more force data accumulates that vision alone could not provide.

Products used

  • FeelFuse®

    A wearable haptic device that returns the sensation of force to the hands in real time. Haptic teleoperation improves both the precision of an operation and the quality of the demonstration data used for imitation learning, and extends the experience in immersive content.

Projects

  • CrafTouch

    Since April 2024, the Japan Kougei Sanchi Association, Ikutouen, Keio University, Nagoya Institute of Technology and commissure have been working on CrafTouch — a project that uses digital technology to record the skills of craftspeople and pass them on to the next generation. It is set at Ikutouen, a Tsuboya ware kiln in Okinawa with 300 years of history, and explores the future of Japan’s traditional crafts industry.

    To record and transmit a craftsperson’s skill digitally, the project needed a practical way to measure the touch felt through the hands and fingers and the force held in the arm. Using SenseFuse®, our wearable haptic sensor, we recorded the movements and sensations that a craftsperson goes through in the ordinary course of making pottery. With FeelFuse®, our wearable haptic display, that recorded force and touch is reproduced on another person’s hand and arm so that it can actually be felt — a system for transmitting the knack and the feel a craftsperson has, without putting it into words.

    We then combined FeelFuse® with a robot avatar for teleoperation to build a teleoperation system with force and tactile feedback. The magnitude of force detected by the force sensor on the robot avatar is mapped in real time to FeelFuse’s rotational shear and vibration stimuli, so that a potter in Okinawa can operate a robot avatar installed in Nagoya while feeling, in their own body, how the force on the object changes.

    Video
    Press coverage
    Publications
    Exhibitions & talks
    Collaborators