Low-Rank Matrix Recovery (LRMR) based salient object detection models decompose image features into a low-rank component associated with background and a sparse component associated with visual salient regions. Then, a saliency map can be generated from the saparse matrix. Existing LRMR-based saliency detection methods are prone to generating scattered or incomplete saliency maps which seldom considers inter-relationship among elements within these two components. A novel LRMR-based saliency detection algorithm is proposed to circumvent this limitation. Firstly, the proposed algorithm generates coarse saliency map by exploiting LRMR-based model that integrates a l1 norm sparsity constraint and a Laplacian regularization term. On the satge of saliency refinement, nonlocal l0 optimization is adopted to improve saliency map quality since it models not only the characteristics of an individual saliency but the interaction between neighbors. Based on the initial saliency map, the nonlocal l0 gradient is utilized to minimize the saliency variation in the salient object efficiently, thereby ensuring the integrity of the salient object. Experimental results on four public datasets of salient object detection validate the proposed algorithm superiority.