Aiming at the unknown gyroscopic factors and drag coefficients of the quadrotor UAV system, which is prone to disturbance by gusts during actual trajectory tracking process, a backstepping control with parameter adaptation and disturbance estimation is designed. Firstly, the mathematical model of the UAV is analyzed, and the model is transformed into a strict feedback format with gusty wind interference. Based on the known parameters and state variable errors, the adaptive learning laws for the gyroscopic factors and the drag coefficients are respectively designed. Secondly, using the parameter values obtained in real time through the adaptive law, and combining the tracking errors of position and attitude as well as their derivatives, the disturbance estimators for the position loop and the attitude loop are respectively designed, the estimated values are used to compensate for the gust disturbances in the control input. Then, a backstepping control algorithm is designed using the adaptive values of the unknown parameters and the estimated values of the gust disturbances. Based on the Lyapunov stability theory, the asymptotic stability of the position loop and attitude loop of the UAV has been rigorously proven. The research results show that the proposed algorithm enables the UAV to precisely follow the desired trajectory and enhances the anti-interference capability when some UAV parameters are unknown and when it is affected by gusts of wind. Finally, the effectiveness of the proposed method is further verified by simulation.