Aiming at the structure design of an echo state network(ESN), an incremental regularized echo state network(IRESN) is proposed in this paper. The reservoir of the IRESN is composed of independent sub-reservoir modular networks. Firstly, the sub-reservoirs are obtained using the singular value decomposition method, and the singular values of the weight matrix of each sub-reservoir can be guaranteed to be less than one. Then, depending on the problem, complexity or residual error, the sub-reservoirs are added to the network one after another until the preset termination conditions are met. In the process of generating the IRESN, the echo state property can be guaranteed without scaling the reservoir weight matrix. Furthermore, in order to tackle the ill-posed problem, in the process of incremental learning, the output weights are trained using the regularization method, and the leave-one-out cross-validation method is used to select the regularization parameter. The simulation results show that the IRESN has compact structure and high prediction accuracy compared with other ESNs.