Aiming at the limited computing and storage environment of smart phones, this paper proposes a lightweight human activity recognition model based on stochastic configuration network with manifold regularization and QR decomposition. Firstly, manifold regularization is used to solve the problem of unpredictable nonlinear distribution after the input data are randomly mapped to the hidden layer space of SCNs, so as to improve the lightweight of the model structure. Secondly, QR decomposition is used to reduce the computational complexity of the output weights and further improve the lightweight of the model modeling process. Finally, based on two human activity data sets, the effectiveness of the proposed model in model recognition accuracy and lightweight is evaluated. Experimental results show that, compared with SCNs, CNN, etc, the proposed model can not only improve the accuracy of activity recognition, but also effectively reduce the calculation complexity and improve the compactness of the model structure.