The goal of offline reinforcement learning is to learn policies from static datasets of previously collected experience. This data-driven learning paradigm greatly expands the potential for applying reinforcement learning in real-world scenarios. However, offline static datasets are often collected from policies of varying quality, leading to a multimodal action distribution that is difficult to model effectively. Furthermore, high-return trajectories are typically scarce within these datasets, which reduces the efficiency of policy learning. To address these challenges, this paper introduces an offline reinforcement learning method based on advantage-constrained diffusion policy. The diffusion model's reverse process is utilized to generate policies that better capture the multimodal distribution. During policy improvement, an advantage function is applied to constrain the policy, directing the agent's focus on the sparse high-return trajectories. Two specific advantage functions are designed for continuous control tasks and sparse reward navigation tasks. Experimental results on bandit tasks and D4RL benchmarks show that the proposed method successfully mitigates limitations in behavior policy expressiveness and the scarcity of high-return trajectories, achieving the highest normalized scores in most tasks.