Based on the criterion function with penalty, the identification problem of controlled autoregressive systems is studied. Through the gradient search, the recursive relationship of parameter estimation is obtained by minimizing the criterion function with penalty, and the optimal step-size is deducted to obtain the parameter estimation through the one-dimensional line search. The projection identification algorithm with penalty and the stochastic gradient identification algorithm with penalty are proposed. By using the multi-innovation theory, a multi-innovation stochastic gradient identification algorithm with penalty is proposed. The effectiveness of the proposed algorithm is verified by the simulation.