What makes Paris look like Paris?

windows with railings in Paris

Cities have their own character. May be that’s what makes some cities notable than the others. In her award winning memoire “Eat, Pray, Love”, Elizebeth Gilbert mentions that there’s a ‘word' for each city.  She assigns ‘Sex' for Rome, ‘Achieve' for NewYork, ‘Conform' for Stockholm (To add more cities that I have been to, how about ‘ Tranquility' for Kyoto, ‘Elegance' for Ginza and ‘Vibrant' for Shibuya?). When terrorists attacked Paris in 2015, more than 7 million people shared their support for Paris under the #PrayforParis hash tag within 10 hours. Have you ever thought what characteristics make a city feels the way it is? Can we make a machine that can ‘feel' the same way about cities as the humans do? 

May be we are not there yet. Nevertheless, researchers from Carnegie Mellon University and Inria have taken an innovative baby step towards this research direction by asking the question “What makes Paris look like Paris?” [1]. Is it the Eiffel tower what makes Paris looks like Paris? How can we find if a given image is taken in Paris if the Eiffel tower is not present in that image? 

To start with, they asked people who have been to Paris before, to recognize Paris from some other cities like London or Prague. Humans could achieve this task with significant level of accuracy. In order to make a machine that can perceive a city as the same way as a human does, first we need to figure out, "What characteristics of Paris help humans to perceive Paris as Paris?".  So, their research focuses on automatically mining the frequently occurring patterns or characteristics (features) that make Paris geographically discriminative than the other cities. Even though, there can be both local and global features, the researchers have focused only on local, high dimensional features. Hence, image patches at different resolutions, represented as HOG+color descriptors are used for the experiments. Image patches are labeled as two sets namely Paris and non-Paris (London, Prague etc.) Initially, the non discriminative patches, things that can occur in any city such as cars or sidewalks, are eliminated using nearest neighborhood algorithm. If an image patch is similar to other image patches in ‘both' Paris set and non-Paris set, then that image patch is considered as not discriminative and vice versa. 

Paris Window painting
by Janis McElmurry

However, the notion of “similarity” can be purely subjective when it comes to similarity between different aspects. So, the standard similarity measurements used in the nearest neighborhood algorithm might not represent the similarity between the elements from different cities well. Accordingly, researchers have come up with a distance or similarity metric that can be learned or adopted to find discriminative features using the available image patches in an iterative manner. This algorithm is executed with images from different cities such as Paris and Barcelona to find distinctive stylist elements of each city.

Interesting fact about this research (well, at least for myself) is artists can use these research findings as useful cues to better capture the style of a given place. More details about this research can be found here.

[1] Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei A. Efros. What Makes Paris Look like Paris? ACM Transactions on Graphics (SIGGRAPH 2012), August 2012, vol. 31, No. 3.