DDoC #02: Discovering Causal Signals in Images

Why?
Modern computer vision techniques are so good at modeling the correlation between the input image pixels and image features. However, if we can perceive the causal structure of a given scene, it helps us to reason better about the real-world.

What?

Discovering Causal Signals in Images (source: https://arxiv.org/abs/1605.08179)

Novel observational discovery technique called "Neural Causation Coefficient (NCC)" to predict the direction of a causal relationship between pair of random variables.  Use NCC to distinguish between object features and context features.

How?

• Causal feature (X > Y): a real world entity (X) that causes the presence of an object (Y) e.g.,  car is there in the presence of a bridge (because, it does not make sense to have a floating car on top of river without a bridge) 
• Anti-causal feature (X < Y): a real world entity (X) that are caused by the presence of an object (Y) e.g., a wheel is there in the presence of a car
• Object feature: Feature that is mostly activated inside the boundary box of an object (e.g., car)
• Context feature: Feature that is mostly activated outside the boundary box of an object (e.g., background) 

NCC is learnt using a synthetic dataset to predict the direction (< or >, e.g., causal or anti-causal) of a given image feature-semantic category (class) pair. Then, the extracted causal and anti-causal features are used to verify if they relate to object features and context features. During their experiments, they have found that object features are mostly related to anti-causal features. Also, they have observed that context feature can either relate to causal or anti-causal features (e.g., road [context]-car vs car-shadow [context]) 

Application: detect object locations in a robust manner regardless of their context 

More information an be found in their paper.