In complex and highly adversarial scenarios such as multi-UAV intelligent aerial combat, simultaneous mastering precise micro-operation decision-making and efficient tactical reasoning is essential for achieving close coordination and gaining dominant advantages. To address the limitations of poor policy generalization and inadequate high-level reasoning capabilities in existing reinforcement learning (RL) methods for such scenarios, this paper introduces a hierarchical decision-making framework integrating large language models (LLMs) with RL (LRHDF). First, inspired by human pilots' decision-making processes, a “LLM-RL” hierarchical framework is constructed, which effectively enhances both low-level micro-operation performance and high-level cognitive reasoning ability. Then, a reflection-based prompt iteration mechanism is implemented, which uses environmental feedback as an optimization signal to optimize prompt instructions continuously. Finally, drawing from human team collaboration, a sequential cooperative decision-making module is developed to explicitly model multi-agent collaboration patterns, thereby improving coordination efficiency. Simulation results and ablation studies on a high-fidelity aerial combat platform demonstrate that the proposed algorithm outperforms traditional RL methods in adversarial performance and generalization across diverse combat scenarios, providing a viable solution for multi-UAV aerial combat challenges.