In this paper, an adaptive tracking controller is designed for the space continuum manipulator in the on-orbit manipulation task. Firstly, considering the modeling error and external disturbance factors, a variable structure controller is designed for the continuum manipulator's dynamic model with typical nonlinear characters. Then, a deep reinforcement learning (DRL) algorithm is adapted to adjust the controller's parameters and to optimize the control performance online. In addition, a sparse training method for the DRL is proposed. In the training process, the origin network's fully-connected layer is replaced with the sparsely-connected layer in a random sparse topology. The weak connections are iteratively pruned in a certain probability, which evolves the DRL's policy network from an initial sparse topology into a scale-free network. Therefore, the network's dimensions are significantly compressed without reducing the training accuracy. Simulation results show that the proposed DRL-based adaptive controller can effectively carry out the tracking control of the continuum manipulator. Through the sparse training method, the quantity of the two hidden layers' parameters is reduced by 99% on the premise of maintenance of the control accuracy. The calculation is effectively reduced.