Automated surface defect inspection in industrial environments is an important aspect of quality management and is of significant research value. Generic detection networks, such as YOLOv4, have proven to be effective in the detection of a wide range of datasets. However, defect detection in industrial environments still needs to address two issues: one is that the percentage of defect instances on the inspected surface is too small, which is a typical small-scale object detection problem; the other is that the structure of generic detection networks is complex and difficult to deploy on mobile devices. To address these problems, this paper proposes a small-scale defect detection method in the industrial environment based on the lightweight deep learning network. Firstly, we replace the YOLOv4 backbone feature extraction network with the GhostNet to improve the feature extraction capability and reduce the complexity of the algorithm. Secondly, the proportion of high-dimensional feature maps in the YOLO head is increased by the improved PANet structure to achieve better performance. The experimental results show that the model can improve the detection accuracy(mAP) by 5.83% while reducing the number of network parameters by 83.5% and improving the detection speed by 2 times, which meets the requirements of accurate and real-time detection.