Cloud and fog contamination in optical remote sensing images causes missing ground information, confusion between cloud shadows and complex land-cover features, and high computational cost in diffusion-based restoration. To address these issues, a multi-scale residual structure-guided diffusion network is proposed. The proposed method takes a single cloudy image and its cloud mask as inputs, and formulates cloud removal as a progressive noise estimation and reverse sampling process. A multi-scale U-Net denoising backbone is adopted, in which spatial attention enhancement is introduced to exploit local contextual information and improve the representation of cloud boundaries, thin-cloud transition regions, and building details. Meanwhile, mask-guided latent feature fusion provides structural priors for occluded regions, and adaptive attention guidance adjusts feature updates in different regions to reduce redundant computation in reliable areas. Experiments are conducted on the Sen2_MTC_New dataset and the self-built City-Building dataset. The results show that the proposed method outperforms several representative cloud removal methods in terms of PSNR, SSIM, and LPIPS. On the City-Building dataset, it achieves a PSNR of 27.6739 dB and an SSIM of 0.8952. These results demonstrate that the proposed method improves structural recovery and detail preservation in cloud-contaminated regions while maintaining favorable inference efficiency.