Inertial Reference Unit (IRU) is an effective means of suppressing the vibration interference of a moving platform’s optoelectronic tracking system. The pointing accuracy of the IRU system is affected by internal and external disturbances such as mechanical resonance, sensor noise, actuator output ripple, and base vibration. The sources and characteristics of different disturbances are different. This paper studies the multi-source heterogeneous disturbance suppression method for the IRU system. Based on the analysis of the sources and characteristics of the internal and external disturbances, a refined disturbance observer (RDOB) structure is proposed to achieve suppression of narrow band large-amplitude disturbances. Combined with the disturbance feedforward structure, the system’s suppression ability against external disturbances is further improved. The effectiveness of the control method is verified through simulation and experiment. The results show that the control structure based on the RDOB and disturbance feedforward technology can achieve a position control bandwidth of 100 Hz, a closed loop pointing accuracy of 1 μrad under multi-source heterogeneous disturbances, and a suppression ability of -32.03 dB against an external disturbance of 20 Hz. Under an external disturbance of 0.1 mrad and 20 Hz, the IRU system has a tracking accuracy of 6.1 μrad fora 0.2 mrad, 1 Hz sinusoidal signal.