Low-rank matrix recovery(LRMR) based salient object detection models decompose image features into a low-rank component associated with background and a sparse component associated with visual salient regions. Then, a saliency map can be generated from the saparse matrix. Existing LRMR-based saliency detection methods are prone to generating scattered or incomplete saliency maps which seldom considers inter-relationship among elements within these two components. A novel LRMR-based saliency detection algorithm is proposed to circumvent this limitation. Firstly, the proposed algorithm generates an initial map by exploiting the LRMR-based model that integrates a ell_1 norm sparsity constraint and a Laplacian regularization term. On the stage of saliency refinement, nonlocal ell_0 optimization is adopted to improve saliency map quality since it models not only the characteristics of an individual saliency but the interaction between neighbors. Based on the initial saliency map, the nonlocal ell_0 gradient is utilized to minimize the saliency variation in the salient object efficiently, thereby ensuring the integrity of the salient object. Experimental results on four public datasets of salient object detection validate the proposed algorithm superiority.