For the problem that the object tracking algorithm using principal components analysis(PCA) has low accuracy in a complex environment, based on the partial least squares analysis, an object tracking algorithm is proposed by using particle filtering with dual models. Firstly, the model of object region is built by the partial least squares analysis, which is applied as the observation model. Then, the dynamic model with dual models is built on Lie group and the corresponding tangent vector space respectively, with the describing the object deformation process by affine transformation. Finally, combining with the update strategy for feature space, the object tracking algorithm is realized by particle filtering. Experiments show that the tracking results are stable and accurate, and the proposed algorithm can effectively filter out the background noise.