The traditional paraphrase identification models based on deep learning usually focus on text representation and ignore the mining and matching of multi-granular interaction features. To address the problem, we propose a recurrent neural network model with word-by-word attention mechanism. In this paper, the word embeddings are inputted into the recurrent neural networks, and the two candidate paraphrase sentences are conditionally encoded via two bidirectional. Based on the output of the iterative hidden states, the sentence-pair representation is obtained from global matching and fine-grained reason via soft-alignment of words and words in the two sentences. Finally, for classification, we use a softmax layer over the output of a non-linear projection of the output vector into the target space of the two classes. The labeled training set for paraphrase identification is small in comparison with the high complexity of the task. In order to make full use of the training data, we use a language modeling task to unsupervised pre-train the neural network parameters on the corpora of more than 580,000 pairs of sentences. This is followed by a fine-tuning stage, where we adapt the model to a specific task with labeled data. Compared with the previous state-of-art neural network model, the accuracy and the $F_1$ score of our model are improved by 2.96 percent and 2 percent on the MSRP data set respectively. The proposed model combines multiple semantic expressions of text from different perspectives and describes the multi-granular matching pattern. It is an end-to-end differentiable system that reduces manual feature engineering efforts, and has good practicability.