In the application of control systems, there is a practical problem that the state cannot be directly measured or the measurement cost is high. In order to solve this problem, a linear discrete-time augmented system with unknown parameters and a state observer is first constructed and the prescribed performance index is defined. Then, based on the separation theorem, the dynamic programming theory and the Q-learning method, a novel off-policy Q-learning algorithm is developed to approximate the optimal observer and the optimal controller for systems with unknown parameters and unmeasured states, such that the control performance is minimized using only measured data. The advantage of this algorithm is that it does not require all the system model parameters to be known and the system state to be directly measurable. Finally, the simulation experiment verifies the effectiveness of the proposed method.