2024 | Tianrui Liu, Qi Cai, Changxin Xu, Bo Hong, Jize Xiong, Yuxin Qiao, Tsungwei Yang
This paper presents a novel approach to image captioning tailored for celebrity photographs, aiming to enhance news industry practices by generating more accurate and contextually relevant captions. The method combines image captioning, face recognition, and noun phrase (NP) chunk matching to produce captions that include specific celebrity names. The system consists of three main components: (1) image captioning using an encoder-decoder architecture to generate general captions without names, (2) face recognition using MTCNN and ResNet to identify celebrity faces and extract their names, and (3) NP chunk matching using NLP techniques to replace generic noun phrases with the identified celebrity names.
The paper discusses the problem definition, including image captioning, face recognition, and celebrity-aware image captioning. It describes the approach, which involves using a CNN and RNN-based encoder-decoder for image captioning, MTCNN for face detection and recognition, and NLP packages for NP chunk matching. The system is evaluated using datasets such as Flickr 8k/30k and COCO Captions, demonstrating high accuracy and effectiveness in generating captions with celebrity names.
The paper also discusses the limitations of the current approach, including mediocre generation performance due to the use of a smaller dataset and the challenges of NP chunk matching in non-exchangeable cases. Potential solutions include using more sophisticated multi-modal approaches, improving dataset quality, and considering the entire task jointly. The study concludes that the proposed pipeline significantly improves the accuracy and relevance of generated content, paving the way for intelligent, automated news generation systems.This paper presents a novel approach to image captioning tailored for celebrity photographs, aiming to enhance news industry practices by generating more accurate and contextually relevant captions. The method combines image captioning, face recognition, and noun phrase (NP) chunk matching to produce captions that include specific celebrity names. The system consists of three main components: (1) image captioning using an encoder-decoder architecture to generate general captions without names, (2) face recognition using MTCNN and ResNet to identify celebrity faces and extract their names, and (3) NP chunk matching using NLP techniques to replace generic noun phrases with the identified celebrity names.
The paper discusses the problem definition, including image captioning, face recognition, and celebrity-aware image captioning. It describes the approach, which involves using a CNN and RNN-based encoder-decoder for image captioning, MTCNN for face detection and recognition, and NLP packages for NP chunk matching. The system is evaluated using datasets such as Flickr 8k/30k and COCO Captions, demonstrating high accuracy and effectiveness in generating captions with celebrity names.
The paper also discusses the limitations of the current approach, including mediocre generation performance due to the use of a smaller dataset and the challenges of NP chunk matching in non-exchangeable cases. Potential solutions include using more sophisticated multi-modal approaches, improving dataset quality, and considering the entire task jointly. The study concludes that the proposed pipeline significantly improves the accuracy and relevance of generated content, paving the way for intelligent, automated news generation systems.