Image sentiment analysis is a hot topic in the computer vision field. However, it is hard to accurately characterize the sentiment semantics by only analyzing the whole image since sentiment judgment is usually subjective. And the image samples with high-quality are scarce. To alleviate the two issues, we propose a multi-head data augment and multi-granularity semantics mining(M²) model. First, a progressive data augmentation model is constructed based on automatic data augmentation and active sample refinement. We improve datasets from the perspectives of quality and quantity. Second, an affective region detection model is introduced for sentiment region augmentation. Intense sentiment semantics is deeply mined from these affective local regions. Then we combine local regions with the whole images to create multi-granularity image data. Third, we pretrain the model through the deep mutual learning framework and affective local regions. The complementary sentiment semantics between heterogeneous SENet networks are fully mined, which is transferred in turn to guide the sentiment analysis of multi-granularity images. Finally, an adaptive feature fusion module is proposed to fuse heterogeneous SENet features to complete multi-granularity semantics mining as well as realize image sentiment analysis. The accuracies of M² are 90.97% and 81.14% on TwitterI and FI, respectively, which outperform mainstream baselines. The M² contains a data augmentation strategy with powerful generalization ability, which builds a firm data basis for training. Meanwhile, the corresponding empirical analysis is satisfactory, indicating a certain practicality.