Aiming at the unknown gyroscopic factors and drag coefficients of the quadrotor UAV system, and it is prone to be disturbed by gusts during the actual trajectory tracking process, a backstepping control with parameter adaptation and disturbance estimation is designed. Firstly, the mathematical model of the UAV is analyzed, the model is transform into a strict feedback format with gusty wind interference. And based on the known parameters and state variable errors, the adaptive learning laws for the gyroscopic factors and the drag coefficients are respectively designed. Secondly, using the parameter values obtained in real time through the adaptive law, and combining the tracking errors of position and attitude as well as their derivatives, the disturbance estimators for the position loop and the attitude loop are respectively designed, the estimated values are used to compensate for the gust disturbances in the control input. Then, a backstepping control algorithm is designed using the adaptive values of the unknown parameters and the estimated values of the gust disturbances. Based on the Lyapunov stability theory, the asymptotic stability of the position loop and attitude loop of UAV has been rigorously proven. The research results show that the proposed algorithm can enable the UAV to precisely follow the desired trajectory and enhance the anti-interference capability of UAV, when some of UAV parameters are unknown and when it is affected by gusts of wind. Finally, the effectiveness of the proposed method was further verified by simulations.