With the development of aerospace technology, space object visual detection as the key methodology of intelligent on-orbit service of spacecraft has garnered broad concerns. Considering the extreme illumination condition and unknown scenario dynamics, the robustness problem of space object visual detection is urged to be studied in depth. This paper proposes a black-box transferred instance attack, which applies adversarial attacks in image classification domain to the task of space object visual detection. It succeeds to fool the EfficientDet model. Meanwhile, we further put forward a cooperative defense strategy that combines adversarial training with the SRMNet denoiser, which effectively enhances the robustness of the object detector. Experimental results show that this defense strategy not only resists adversarial attacks successfully, but also makes a great improvement on the accuracy of the space object detection model.