For the problem that the target tracking algorithm based on Mean Shift may be lost when the infrared target has low SNR or owns a dynamic change in scale, an improved Mean Shift tracking algorithm is proposed. Firstly, the features of gray and texture are fused to enhance the target information. Then coefficients based on the gray and texture histograms of the background pixels around the target are computed and incorporated into the computation of gray and texture histograms. After the accurate localization of the infrared target obtained, an objective function based on the similarity of background and target is proposed. Finally, the adaptive bandwidth is obtained through making the objective function minimum. Experimental results show that the proposed algorithm is effectiv and robust and can be adapted to the target change in scale.