This study investigates the design of a hover attitude controller of a flying-wing tail-sitter unmanned aerial vehicle(UAV) by taking into account the uncertain impacts such as actuator failure, wind field disturbance, aerodynamic uncertainty and the uncertainty of moment of inertia. Based on the linearized kinematic and dynamic model near the hovering point, a robust servomechanism linear quadratic regulator(RSLQR) controller is designed to ensure the good response and robustness of the nominal system. Meanwhile, considering the performance degradation of the controller in exist of large uncertainties and disturbances, the $L_1$ adaptive controller is designed to compensate for the uncertainties and disturbances so that the aircraft could operate near a large range of the equilibrium point. Considering that the time delay margin of controller is crucial to the stability of system, the relationships between the time delay margin and the controller parameter are discussed. Simulation results show the good performance in exist of different uncertainties and a method of compensation based on the extended state observer(ESO) is proposed to ensure good response when the control system is applied to a performance-limited hardware. Finally, a flight test is implemented to verify the validity and feasibility of the algorithm.