SMGT: A self-supervised multimodal graph transformer with coverage-aware selection for video summarization with nanotechnology-inspired nanoscale representation learning

Abstract

Video summarization is essential for efficient video browsing, retrieval, and analysis by condensing long content into informative summaries. However, current methods face challenges such as reliance on costly annotations, limited datasets, weak multimodal fusion, and structural limitations. The proposed SMGT framework addresses these issues by learning from raw multimodal data without using summary labels. It constructs a frame-aligned graph and applies a graph-transformer to capture dependencies, followed by calibration and coverage-aware selection. Experimental results show competitive performance, achieving 50.33% on TVSum and 44.31% on SumMe, demonstrating scalability and effectiveness in low-label environments. Furthermore, nanotechnology-inspired approaches provide a novel perspective for modeling information at the nanoscale level, enabling fine-grained feature representation, improved multimodal fusion, and enhanced structural learning. These nanoscale-inspired representations can improve the precision of frame importance estimation and support more efficient and scalable video summarization frameworks.