For the design of the neural network architecture, a pruning algorithm based on the neural complexity is proposed. The essence is to calculate the entropy of neural network by the standard covariance matrix of the neural network’s connection matrix in the process of training, and the network’s complexity can be acquired. In the premise of ensuring the information processing capacity of neural network, the least important hidden node is deleted. It is not necessary to train the cost function of the neural network to a local minimal, suitable for pruning neural network architecture on-line, and the pre-processing neural network weights are avoided before architecture adjustment of the neural network. The simulation results of the typical function approximation show that the precision of the approximation is ensured and at the same time a simple architecture of
neural networks can be achieved.