Activating demand-side flexible resources, such as building thermal loads and electric vehicles, and promoting bidirectional supply-demand interaction are key to improving the economic performance and operational flexibility of integrated energy microgrids (IEMs). However, traditional centralized optimization methods struggle to balance privacy protection and decision-making autonomy, while existing multi-agent deep reinforcement learning (MADRL) methods face coordination difficulties caused by information isolation and insufficient guarantees of physical constraint satisfaction in complex multi-energy coupling scenarios. To address these challenges, an improved MADRL-based coordinated supply-demand dispatch method is proposed. First, a dual-agent collaborative decision-making model comprising supply-side and demand-side agents is established to coordinate system economy and user comfort while preserving the decision-making autonomy of each agent. Second, under the centralized training and decentralized execution framework, a cross-attention-based centralized critic network is developed to enhance the perception of cross-agent coupling features under information isolation and improve cooperative learning efficiency. Finally, a safe action projection mechanism based on system physical boundaries is introduced to constrain policy outputs within the feasible action set, thereby improving policy safety and practical implementability. Case studies demonstrate that the proposed method reduces system operating costs, improves user comfort, and satisfies physical constraints during both training and operation while maintaining decentralized decision-making autonomy. Compared with benchmark methods, it achieves better performance in terms of convergence, economic efficiency, user comfort, and constraint satisfaction.