Combined Depth and Semantic Segmentation from Synthetic Data and a W-Net Architecture

Abstract

Monocular pixel level depth estimation requires an algorithm to label every pixel in an image with its estimated distance from the camera. The task is more challenging than binocular depth estimation, where two cameras fixed a small distance apart are used. Algorithms that combine depth estimation with pixel level semantic segmentation show improved performance but present the practical challenge of requiring a dataset that is annotated at pixel level with both class labels and depth values. This paper presents a new convolutional neural network architecture capable of simultaneous monocular depth estimation and semantic segmentation and shows how synthetic data generated using computer games technology can be used to train such models. The algorithm performs at over 98% accuracy on the segmentation task and 88% on the depth estimation task.