In order to solve the tracking problem of video sequences in real-world scenarios, a robust tracking algorithm based on adaptive feature selection is proposed. Firstly, for the problem that the candidate features of the online AdaBoost algorithm are not robust, a construction mode of the candidate feature pool is proposed, which combines color and pyramid gradient orientation histogram features. Then, for the problem that classifiers are vulnerable to the influence of improper samples during the update, a process of occlusion detection is added at each frame after obtaining the tracking result to avoid the phenomena of drift. Lots of comparison experiments show that the proposed algorithm tracks the object accurately and reliably in realistic videos.