In order to optimize the spatial and spectral details of the fused image, this paper proposes a double second-order attention spatial and spectral details compensation full convolutional network (FCN). The network extracts spectral features and spatial features from hyperspectral images and multispectral images, respectively, and fuses the dual-mode features. At the same time, it pays attention to reconstruct the fused features into the required hyperspectral images with high spatial resolution under the constraint of detail compensation. The double second-order attention residual module proposed focuses on extracting the spatial details and channel details of the image. The channel structure feature is extracted by the second-order statistics of the channel gradient representing the channel relationship. The high-frequency details of the image are captured by the second-order statistics of the spatial relationship represented by the physically interpretable image structure tensor. The Laplace loss and spectral angle mapping loss are added to the loss function to further improve the texture and spectral similarity between the fused image and the reference image. The fusion experiments are carried out on two sets of simulated data sets, and compared with many methods, and the quality of the fused image is indirectly verified from the perspective of classification. The results show that the fusion image of this method has the best performance in each index. The classification accuracy of fused images can indirectly reflect the good fusion effect of this network. Experiments on two real datasets further verify the good generalization ability of the proposed method.