Aerial manipulator shows intense research value and application value in terms of interacting with the external environment. However, the improvement of operation ability of the rotary-wing flight robot is restricted at present, which is caused by the weak pose control performance, insufficient load capacity, and short endurance time. This paper designs a novel cable-driven aerial manipulator system. Considering the flexibility of cable-driven mechanism, a rigid-flexible coupling dynamics model has been established considering the joint flexibility. To address the trajectory tracking control issue in joint space under lumped disturbances, a linear extended state observer is adopted to estimate and compensate the lumped disturbances. Meanwhile, a super-twisting algorithm and fractional-order non-singular terminal sliding mode are used to ensure good control performance in both the reaching phase and the sliding mode phase. Furthermore, the stability of the proposed controller is proven with the Lyapunov theory. Finally, the effectiveness of the proposed controller is verified through visual simulation and ground test. All the results show that the proposed controller has faster response speed, stronger disturbance rejection ability, and higher tracking accuracy compared to other common controllers, which can satisfy the control requirements of the cable-driven aerial manipulator.