For the tracking problem when the target undergoes rapid and significant appearance changes, a novel tracking algorithm is presented. The object’s appearance is represented by a set of local patches with inherent spatial geometric constraints relationship. It probabilistically adapts to the object’s appearance changes by removing and adding the local patches. The locations of new patches are determined by the global color property，which can improve the limitations of the traditional patch-based algorithms that the appearance model can’t be updated in time during tracking. Experimental results show that the proposed algorithm performs in many cases with high adaptivity to appearance changes，which has high accuracy to objects with drastically changes.