To enhance safety in the control process of robotic manipulators, an adaptive reinforcement learning-based control method is proposed for addressing error constraints in manipulator systems. For this purpose, the dynamic system of the manipulator is transformed into a dynamic equation concerning tracking errors. Then, an error transformation function is employed to convert the constrained error system into a new unconstrained system, based on which an optimal controller is designed. To solve the optimal control problem, a reinforcement learning approach is used to approximate the solution of the Hamilton-Jacobi-Bellman (HJB) equation, where a critic network approximates the optimal value function and an actor network approximates the output of the optimal controller. A positive definite function is introduced to significantly simplify the adaptation laws of the critic and actor networks. Based on Lyapunov stability theory, all error signals of the system are proven to be semi-globally uniformly ultimately bounded. Finally, the effectiveness of the proposed method is verified through a simulation example of a two-degree-of-freedom robotic manipulator.