Particle filter is introduced into the adaptive mean shift tracking algorithm, and a new object tracking method is proposed, which tracks object scale with a dynamic band-matrix. It combines the detection results to determine the object location with a weighted summation method, and avoids the system from falling into local optimum. All the particles are converged and re-sampled in a place near the precisely determined location, keeping the diversity with fewer particles and reducing the accumulated error. An extended searching strategy is proposed to be used in target re-search once it is lost. Experiment results show that the proposed method is robust in complex environment and the tracking trajectory is smooth.