Publication

Abstract : Automatically generating a characteristic language portrayal of an image has pulled in interests in light of its significance in practical applications and on the grounds that it associates two significant artificial intelligence fields: natural language processing and computer vision. This paper proposes a partially supervised model for generating image descriptions based on instance segmentation labels. Instance segmentation, a combined approach of object detection and semantic segmentation is used for generating instance level labels which is then used for generating natural language descriptions for the image. The instance segmentation model uses MRCNN framework with feature pyramid networks and region proposal networks for object detection, and fully convolution layer for semantic segmentation. Information obtained from different local region proposals are used to generate region wise captions. Important aspects of the caption include distance, color and region calculations based on the results obtained from the instance segmentation layers. This paper uses instance segmentation layer information such as ROIs, class labels, probability scores and segmentation values for generating effective captions for the image. The proposed model is evaluated on Cityscape dataset where the primary objective is to provide semantic scene understanding based on the instances available in urban areas.