Complex aerial combat scenarios face two major challenges: 1) low online sampling efficiency with potentially catastrophic high-risk exploration; 2) scarce offline data with limited coverage that undermines generalization to out-of-distribution states. To address these challenges, we propose CQL-Safe, a risk-constrained offline algorithm built on conservative value function learning, where risk is defined as the probability of agent damage or shoot-down and related safety-threat metrics. CQL-Safe constructs an augmented offline dataset combining learning from demonstration and diffusion model to alleviate data scarcity, embeds a multi-dimensional risk evaluation function into the conservative value function learning framework to curb overestimation of out-of-distribution actions, and adopts a Lagrangian multiplier to adaptively tune constraint strength, achieving a practical balance between reward maximization and safety. Experiments across diverse aerial combat scenarios demonstrate improved training efficiency and superior performance, substantially enhancing policy effectiveness and generalization while maintaining agent safety.