Transformer has achieved excellent results on large-scale data sets, but it is too complex due to utilizing Multi Head Attention(MHA), and its performance is poor on small-scale data sets. The study on the replacement of MHA is little in the field of natural language processing, although it has made great achievements in the field of image processing. Therefore, a method called multi-layer semantics perceptron(MSP) is proposed. Its major innovation is that instead of MHA, a simple token sequence transformation function is used, thus achieving a better semantic feature representation with lower complexity. Additionally, a dynamic depth control framework (DDCF) is proposed, which is able to optimize the depth of neural networks automatically, as a result the complexity of the model is reduced markedly. Finally, based on the MSP and the DDCF, the dynamic multi-layer semantics perceptron model (DMSP) is proposed. Compared with the Transformer model with same depth, the experimental results on multi-data sets show that the DMSP model achieves better performance significantly, meanwhile, its parameters declines sharply.