Aerial manipulator shows intense research value and application value in terms of interacting with the external environment. However, the improvement of operation ability of the rotary-wing flight robot is restricted at present, which the primary reasons are the weak pose control performance, insufficient load capacity, and short endurance time. This paper designed a novel cable-driven aerial manipulator system. Considering the flexibility of cable-driven mechanism, a rigid-flexible coupling dynamics model has been established since considering the joint flexibility. To address the trajectory tracking control issue in joint space under lumped disturbances, a linear extended state observer is adopted to estimate and compensate the lumped disturbances. Meanwhile, a super-twisting algorithm and fractional-order non-singular terminal sliding mode are used to ensure good control performance in both the reaching phase and the sliding mode phase. Furthermore, the stability of the proposed controller has been proven with Lyapunov theory. Lastly, the effectiveness of the proposed controller has been verified through visual simulation and ground test. All the results show that the proposed controller has faster response speed, stronger disturbance rejection ability, and higher tracking accuracy compared to other common controllers, which can satisfy the control requirements of the cable-driven aerial manipulator.