Based on the parallelization capability of the simple recurrent unit(SRU) network and the connection strategy of highway-networks, this paper proposes a highway-simple recurrent unit(H-SRU). The H-SRU replaces the activation function of the cell state with the non-saturated activation function to effectively solve the vanishing gradient problems. Additionally, it introduces the idea of feed-forward link used in highway-networks into the cell state representation of the SRU to make the network more sensitive to gradient changes. Natural language processing models are built to verify the effectiveness of the proposed method using the PTB (Penn treebank dataset) and WikiText-2 data sets. The results show that the proposed H-SRU dramatically improves the performance of recognition, while maintaining high training speed. The perplexity value of the H-SRU on the WikiText-2 data set reaches 26.1, which is currently the best known, and its efficiency is higher than that of non-SRU networks.