In order to solve the problem of low detection accuracy of moving small targets in the panoramic video in complex background, a small target detection algorithm based on complex background motion is proposed. Firstly, to reduce the interference of complex background information and improve the accuracy of target detection, the fast robust principal component analysis(Fast RPCA) algorithm is used to separate the foreground background information of the panoramic video image, and the foreground information is extracted as an effective image feature. Then, the candidate frame size of the region proposal network(RPN) in the faster region-convolutional neural networks(Faster R-CNN) is improved to adapt to the target size in the panoramic image, and then the foreground feature map is trained. Finally, the convolutional layer output detection model is shared by the RPN network and the Fast R-CNN network to achieve accurate detection of small targets in the panoramic video image. Experiments show that the proposed algorithm can effectively suppress the influence of complex background information on target detection accuracy, and has high detection accuracy for small moving targets in panoramic video images.