With the rapid development of unmanned system platforms, unmanned swarm confrontation has become crucial for the future development and construction of system capabilities. However, when engaging swarm targets, adversarial interactions between swarms exacerbate issues such as communication bandwidth pressure, dynamic link variations, and accumulated information delays under limited communication constraints. Swarm confrontation decision-making faces challenges including overloaded situational information, degraded collaborative capabilities, and delayed team responses. To address these problems, this study first establishes kinematic models and attrition models for multi-agent confrontation, and subsequently proposes a team game modeling and solution approach for multi-agent confrontation under limited communication constraints. By designing a team alliance strategy based on small-world network communication, limited communication conditions are incorporated into the team game model for multi-agent confrontation. The definition and existence conditions of team game equilibrium are provided, based on the concept of correlated team max-min equilibrium solutions. Then, a heterogeneous multi-agent reinforcement learning algorithm is developed to design an alternating freeze-training method for solving the team game equilibrium. Finally, the simulation results demonstrate that the proposed game model and alternating freezing training framework can obtain stable strategy profiles, achieving adaptive dynamic decision-making and adversarial strategy optimization under limited communication conditions.