Robust Change Captioning in Remote Sensing: SECOND-CC Dataset and MModalCC Framework

Abstract

Existing remote sensing image change captioning (RSICC) methods often fail under challenges, such as illumination differences, viewpoint changes, and blur effects, leading to inaccuracies, especially in no-change regions. Moreover, images acquired at different spatial resolutions and with registration errors tend to affect the captions. To address these issues, we introduce SECOND-CC, a novel RSICC dataset featuring high-resolution RGB image pairs, semantic segmentation maps, and diverse real-world scenarios. SECOND-CC contains 6041 pairs of bitemporal remote sensing images and 30 205 sentences describing the differences between the images. In addition, we propose MModalCC, a multimodal framework that integrates semantic and visual data using advanced attention mechanisms, including cross-modal cross attention and multimodal gated cross attention. In addition, we adapt MModalCC to handle noisy semantic inputs by integrating a semantic change detector, improving its robustness for real-world applications. Detailed ablation studies and attention visualizations further demonstrate its effectiveness and ability to address the challenges of RSICC. Comprehensive experiments show that MModalCC outperforms state-of-the-art RSICC methods, including RSICCformer, Chg2Cap, and PSNet with +4.6% improvement on BLEU4 score and +9.6% improvement on CIDEr score in SECOND-CC dataset. MModalCC was further validated on the LEVIR-MCI benchmark, where it achieved an average S*m score of 83.51, significantly outperforming previous state-of-the-art methods.