Panoptic segmentation is a topic that was discussed during our lab seminar recently, because it could potentially improve scene understanding in autonomous vehicles using vision sensors.
Successful approaches based on convolutional nets have been proposed previously for semantic segmentation task. Further, methods based on object or region proposals have become popular to detect individual objects as well.
Image source: [1]
The idea behind panoptic segmentation [1] is unifying the tasks of semantic segmentation (studying about 'stuff' such as sky, grass, regions) and instance segmentation using object detectors (studying about countable 'things', E.g., different instances of cars).
'Panoptic quality (PQ)' metric is proposed as a novel method to evaluate the proposed approach. More details about this can be found here and a simpler version here.
[1] Panoptic Segmentation, Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, Piotr Dollar
Information theory has been proved useful in many applications such as digital communication, computer networking and multimedia compression. For example, the concept of entropy (average information content) can be used to determine the minimum length a given message can be encoded before transmission and the concept of mutual information can be used to determine the channel capacity of a given communication channel.
Information bottleneck (analogous to optimal representations)
Regardless of the recent breakthroughs of deep learning, 'why they work, the way they work?', still remains a mystery. Recently, I read about an interesting attempt [1] where the researchers tried to understand the inner workings of deep learning algorithms using the notion of information theory. There, they have come up with a concept called "information bottleneck" to describe the most relevant features or signals with sufficient discriminative ability, excluding all the noise, to solve a given problem. In other words, the goal of the deep neural network is to maximize the information bottleneck (compressing the network as much as possible while preserving the prediction accuracy).
More information can be found here (their original paper) and here (a much simplified version!).
[1] Opening the Black Box of Deep Neural Networks via Information, Schwartz-Viz & Tishby, ICRI-CI 2017
This year, I presented my research "Multi-Agent Based Road Traffic Control Optimization" in Intelligent Transportation Systems Conference (ITSC 2017), International workshop on Large-Scale Traffic Modeling and Management, Yokohoma. I worked as a student volunteer there as well.
Connected Vehicle
Key highlight of the all three keynote speeches is the concept of "Connected Vehicle". Different V2X technologies to connect vehicles, infrastructure and people to provide better mobility services were discussed. First keynote was "SIP Autonomous Driving" delivered by Mr. Masao Fukushima (Nissan R&D). Development of a digital dynamic map platform with the aim of realizing autonomous driving and advanced driver assistance systems was discussed. Dynamic map is an essential component in autonomous car navigation. This is done by Dynamic map platform Pvt Ltd.
ART information center was introduced as a way of gathering required mobility information such as traffic congestion, waiting time and bus route information. Currently, they are planning to come up with map using point cloud information and IoT based services will be added in the future.
Next Keynote was "From the World's 1st Car Navigation, towards Connected and Automated Driving in the Future" by Mr. Yoichi Sugimoto (Honda R&D). This was an interesting session where he explained the "Honda electro gyrocator", world's first commercialized map based navigation system (1981).
He further spoke about more advancements innovated by Honda R&D to realize connected vehicle concept. One such innovation is "Honda Telematics services"
(1998) that focuses on leveraging IoT solutions to provide connected
car platform.
Further, they have introduced Internavi system route, the
world's first floating car system that collects traffic information from vehicles and provide them to it's users.
The next Keynote speech was done by Prof. Emeritus U. Ozguner (Control & Intelligent Transportation Lab - Ohio State University). The topic was "Smart cities: An Intelligent Vehicles Perspective". One highlight of the session was "Smart Columbus", an experimental smart city that is designed to enable the connected vehicle concept.
It's another exclusive day at Miraikan! We got free passes to Miraikan special exhibition "Art of Disney - The Magic of Animation" (Thanks to TIEC #TIECrox! :D).
Initially, we were under the impression that we are going to see lot of cartoon sketches and not more than that. However, soon we realized that we were wrong. Walt Disney Studios have demonstrated their journey from 1923 to 2017 in a way that we were so amazed on the sheer effort and attention to detail they have placed to keep the innovation alive in each and every production they embarked upon. Here's few examples that impressed me.
Pinocchio - They've used multi plane cameras to add dimensionality (depth or 3D effects) as a visual effect
Bambi - They have studied animal anatomy (of deers) and they have used live animals as reference (There has been few deers, so that artists can observe their moves and behavior), Further, they have used minimalistic ink to depict forests.
"Always as you travel assimilate the sounds and sights of the world" - Walt Disney
Saludos Amigos - Before they produce animated films in diverse range of settings, they observe locations, societies, cultures, prominent shapes and colors during field visits to influence the "feel" of the final work. Observing the unique South American colors for "Saludos Amigos" is one such example. Similarly, they have observed Japanese culture for "Big Hero 6" and Africa for "Jungle book"
Fantasmic - Creative visual effects itself won't make the experience of the audience complete as it would address only one human sense. In Fantasmic, they have introduced the concept of visualization of sounds of classical music.
Dumbo - Dumbo is an elephant who doesn't talk. So, they have used effective expressions of emotions to convey its feelings to the audience.
Sad dumbo
Happy Dumbo with Opened ears and bright eyes
Lady and the Tramp - In Lady and the tramp, scenes are viewed as how a dog sees the world. (Few centimeters above the ground. (Dog's eye view)
How a dog sees the world? - A Dog's eye view scene
Frozen - Physical properties of snow (snow effects) has been considered in the animation movie for snow simulation in scenes (As given in the video below)
Zootopia - They have analyzed animal hair and fur in different animal parks to get realistic look for their own animal characters.
Animal fur reference. Source: https://www.fxguide.com/featured/the-fur-reaching-tech-of-zootopia/
101 Dalmations - In this animated movie, they have used Xerox copying technology to animate many similar looking dogs. More information on that here.
Now a days, animations can be developed vastly with advanced computer graphics technologies and Disney Studios continue to strive on pushing the boundaries of imaginations as they used to be!
Having read the profound
master piece “When breath becomes air”, by Neuroscientist – surgeon Paul
Kalanithi, I was curious about how neuroscience could contribute to AI
(Computer vision in particular).
Then, I found an comprehensive article in Neuron Review
journal (written by Demis Hassabis, Dharshan Kumaran, Christopher Summerfield,
Matthew Botvinick) titled “Neuroscience inspired Artificial Intelligence”. Here goes a brief excerpt of concepts I found
inspiring in that article, related to computer vision.
Past
CNNS
How visual input is filtered and pooled into simple and
complex areas of cells in area V1in visual cortex
Hierarchical organization of mammalian cortical
systems
Object recognition
Transforming raw visual input into increasingly complex set
of features - To achieve invariance towards pose, illumination and scale
Present
Attention
Visual attention shifts strategically among different
objects (no equal priority for all objects) - To ignore irrelevant objects in a
given scene in the presence of a clutter, multi object recognition, image to
caption generation, generative models to synthasize images
Future
Intuitive understanding of physical world
Interpret and reason about scenes by decomposing them into
individual objects and their relations
Redundency reduction (encourages the emergence of
disentangled representations of independent factors such as shape and position)
- To learn objectness, construct rich object models from raw inputs using deep
generative models, E.g., Variational auto encoder
Efficient Learning
Rapidly learn new concepts from only a handful of examples
(Related with Animal learning, developmental psychology)
Characters challenge - distinguish novel instances of an
unfamiliar hand written character from another - "Learn to learn” networks
Transfer Learning
Generalizing or transferring generalized knowledge gained in
one context to novel previously unseen domains (E.g., Human who can drive a car
drives an unfamiliar vehicle) - Progressive networks
Neural coding using Grid codes in Mammalian entorhinal
cortex - To formulate conceptual representations that code abstract, relational
information among patterns of inputs (not just invariant features)
Virtual brain analytics
Increase the interpretability of AI computations, Determine
response properties of units in a neural networks
Activity maximization - To generate synthetic images by
maximizing the activity of certain classes of unit
From AI to neuroscience
Enhancing performances of CNNs has also yielded new insights
into the nature of neural representations in high-level visual areas. E.g., 30 network architectures from AI to explain the
structure of the neural representations observed in the ventral visual stream
of humans and monkeys
Some insightful concepts I learnt related to the roots of innovation during our visit to Miraikan - The future museum in Odaiba (The best science museum I've ever visited so far and I'm so glad it's just there in our neighborhood. One day is definitely not enough to completely explore this place.)
Association - Associating novel concept from one field for the advancement of another field, Conventional computer vs Quantum computer (Associating the properties of quantum mechanics with computer science)
Examples:
Quantum dot marking (Associating placing an marker on an object to detect target substances on the smallest scale)
Intra body communications (Associating conductivity with telecommunications with communicating via the human body
Integration - Combining and integrating things with different properties for a single purpose gives us the ability to generate new things (The idea of lab on a chip)
Examples:
Bio machine hybrid system (Insect controlled robot to investigate the ability of an insect to adopt to perturbations
Mechano bionic machine (Integrating living muscles as a power source to machines) - Power source coming from the heart of an insect
Metal plated fibres (Make fabric conductive by plating the surface of a synthetic fibre with a metal - lightness, strength, flexibility along with conductivity) - Applications in Electronics products
Serendipity - Unexpected developments give us the ability to make fortunate discoveries - The idea of conductive polymer (conductive plastics) by Dr. Shirakawa
Examples:
Post-it notes (Easily attachable and detachable memo slips as a solution for falling book marks using low tack adhesive)
Hook and loop fasteners - idea inspired by the pet dog afflicted with burrs
Large scale synthesis of carbon nano tubes
Mimic - Taking hints from the existing functions and forms gives us the ability to create things that formally didn’t exist or achieve things that couldn’t be done - Artificial Photo synthesis (Bio inspired)
Examples:
Learning super water repellency from lotus leaves
Morphotex - development of fibre that generates beautiful colors without dying inspired by the wings of morpho butterfly
Alternative- New ideas unconstrained by traditional values give us the ability to create new things (Color filter for a LCD)
Examples:
Making artificial skin using the hair thrown away during a hair cut (self recycling)
Retinal imaging display (project a video directly into retina in the eye) same as pouring music using ear phones)
Power generating floor (using the force applied to the floor while walking)
