In order to solve the problems of less-prominent features and low image contrast in infrared and visible image fusion, the paper proposes a hybrid model-driven fusion algorithm. Firstly, the latent low-rank representation model is used to extract the base sub-band, significant sub-band and sparse noise sub-band of the infrared and visible image respectively. Then, the base sub-band is decomposed into low frequency coefficients and high frequency coefficients by the non-subsampled Shearlet transform model. The low frequency coefficients are accurately fitted by dictionary learning and sparse representation, while the high frequency coefficients are selected by local window combined with logic weighting. The significant sub-band is fused by adaptive weighting of regional energy ratio threshold. Finally, the fused low frequency coefficients and high frequency coefficients are $1^st$-order reconstructed to obtain the fused base sub-band and discard the sparse noise sub-band. Then the $2^nd$-order reconstruction is conducted with the fused significant sub-band to get the fused image. The experiment results show that the proposed algorithm can obtain clear fused image with rich information. The fused results have high contrast and significant target contour which can improve the recognition of the scene.