Facing the detection requirements of multi category and variable scale vehicles in the road traffic scene, how to effectively construct structured data with low computational power to achieve beyond sight distance perception, and overcome the limitation of single vehicle sight distance is one of the important problems to be solved in the field of autonomous vehicle environment perception technology. In this paper, we propose a lightweight roadside perspective based multi object detection algorithm that balances accuracy and real-time performance. First, a reverse residual network structure embedded in the channel domain attention mechanism is used as the backbone of the network, replacing the single stage detection algorithm feature extraction network with a deep separable convolution to reduce the number of feature extraction network parameters. Second, spatial pyramid pooling(SPP) is used to process the output feature map of deep networks, then we select maps of different depth feature in the lightweight backbone network to output, and use the path aggregation network(PANet) to fuse deep semantic information and shallow superficial information to form the neck of the detection model. Finally, at appropriate network depth, three different network outputs of feature map sizes are set at the head of the detection model to regress the target information of different sizes of targets in the same image. A lightweight detection model M3-YOLOv4 is established. The experimental results show that the mAP of M3-YOLOv4 on RS-UA dataset is 0.906, which performs 1.1% decrease compared to the YOLOv4. The parameter quantity of the M3-YOLOv4 model is reduced to 10% of the YOLOv4, and the forward inference speed of the model on the same platform also shows significant advantages.