Image sentiment analysis is a research focus in the field of computer vision. However, we are faced with the following key problems: First, owing to the subjective differences of different annotators, high-quality samples with definite sentimental annotations are very scarce. Second, the implicit cross-modal semantics among heterogeneous features has not been fully explored. To address these two problems, we propose an active sample refinement & feature fusion (ASRF²) via active sample refinement and cross-modal semantics mining: an active sample refinement strategy is designed by fusing the active learning and sample refinement ideas. High-quality samples with definite sentimental annotations are obtained in turn. Then, the state-of-the-art discriminant correlation analysis (DCA) algorithm is employed to fully mine the cross-modal correlations among the heterogeneous features. Low-dimensional but more discriminant cross-modal semantics that can better depict the key sentimental contents of images are generated. The cross-modal semantics is used to train a Catboost classifier and complete image sentiment analysis. We validate the proposed ASRF² model on the TwitterI and FI datasets. The corresponding accuracies reach about 90.06% and 75.77%, respectively, which outperform other state-of-the-art baselines as well as the real-time efficiency. Compared with the baselines, the proposed model only needs two image features, and it is easy to tune and reproduced the ASRF² model. Moreover, the ASR strategy is robust, which can offer many more high-quality samples for the baselines to improve the final recognition performance.