The current infrared and visible image fusion methods based on deep learning usually cannot perceive the saliency region of source images. As a result, the result of image fusion fails to highlight the respective typical features of infrared and visible images, and the ideal fusion effect cannot be reached. In response to these issues, an improved residual dense generative adversarial network structure suitable for infrared and visible image fusion tasks is designed. First of all, the improved residual dense block is used as the basic network to construct the generator and the discriminator respectively; the squeeze and excitation network based on attention mechanism is introduced to capture the salient features in the channel dimension, which will preserve the thermal radiation information of infrared images and the textures information of visible images adequately. Then, the relativistic average discriminator is used to measure the relative average difference between the fusion image and two source images respectively, so as to instruct the generator to preserve the missing information of source images based on the difference. Finally, the experimental results on multiple image fusion datasets such as the TNO, prove that the proposed method can generate fused images with clear targets and rich details. Compared with the fusion method based on the residual network, the edge intensity and average gradient are increased by 64.56$%$ and 64.94$%$, respectively.