DDoC #04: Connecting Look and Feel

Today's paper is "Connecting Look and Feel: Associating the Visual and Tactile Properties of Physical Materials". (Figure 1: with a self pat on my back for working on this, even when I'm little sick+in a bad mood. And then I found this cool image!)

Figure 1: self pat on my back source:
http://massagebywil.com/2011/10/25/pat-yourself-on-the-back/

Why?

Humans use visual cues to infer material properties of objects. Further, touch is an important way of perception for both robots and humans to effectively interact with the outside world. 

What?

Project the input from different modalities in to a shared embedding space to associate visual and tactile information.  

How? 

Fabrics are clustered to different groups using K-nearest neighbor algorithm based on their physical properties such as thickness, stiffness, stretchiness and density. For humans, these fabrics in similar cluster will have similar properties. 

Clusters of Fabrics with different properties 

Input: Different modalities of input image of fabric (depth, color and tactile images from touch sensor) 

Output: Determine the whether the different modalities are from same fabric or different fabrics 

Process: 

First, a low dimension representation (different embedding) of these input data is extracted using CNN. Then the distance between these different embeddings is measured. The idea is to have smaller distance for different modalities of the same fabric and to have a larger distance for different modalities of the different fabric.  

So, the goal of optimization function is to minimize the distance between different modalities of same fabric using contrastive loss (In layman terms, neighbors are pulled together and non-neighbors are pulled apart)  

More information can be found in their paper.