When the aircraft actuator fails, there are a variety of internal unmodeled dynamics, system uncertain parameters and unknown input gain. Therefore, a $L_1$ adaptive fault tolerant control method for the multi-input multi-output aircraft actuator fault system with matched and unmatched uncertainties is proposed. Firstly, the equivalent(semi-) linear time-varying system is derived. Then, based on the projection operator, a $L_1$ adaptive fault tolerant controller is proposed. The Lyapunov equation is derived, and stability is proved. Finally, the transient and steady state performance of the controller are analyzed. The simulation results show the good fault tolerance performance, robustness and stability of the proposed method, meanwhile, the transient and steady state boundedness of system output, control signal and parameter estimation are ensured.