A novel algorithm of object model update for visual tracking is presented. Firstly, feature histograms combined with spatial information are used to model the object and background. Then, for each feature space, the cross entropy measure of information theoretic is applied to evaluate the divergence between the distributions of object and background, and the feature space with maxi-mal divergence is selected to update the object model. Thereafter, the likelihood map for the next frame is constructed according to the current updated models. The peak of the map, which is given by mean shift, is thought as the new position of the object. To alle-viate the tracking drift problem, we incorporate the model updating process with the figure/ground segmentation, which is based on systematic integration of spatial and temporal data over time using conditional random field (CRF). The analyses and experiments show that the proposed algorithm can track targets well under clutter, scale variations, and partial occlusions.