Because of the problems of noise interference, illumination changes, and target occlusion of the visual monitoring in the factory, the existing fall detection algorithms have weak anti-noise ability, high lighting requirements, and poor target detection effect. It is intractable for the state-of-the-art visual surveillance methods to be reliably applied to all-weather factory workshop worker safety monitoring scenarios. Therefore, this paper presents a novel human fall detection method based on multimodal visual monitoring. Firstly, a thermal imager and a visible light camera are used to obtain the monitoring images in the workshop, and an adaptive median filter model is proposed to denoise the images and to suppress the interference of environmental noise on the monitoring images. Secondly, an improved lightweight convolutional neural network is used to extract the worker skeleton and joint sequence, and a time frame merging module and a pose residual fusion module are designed, so that the network can ensure that the worker pose occluded can be detected through the temporal correlation of adjacent frames. Finally, the inclination of the human body axis, the aspect ratio of the human body circumscribed rectangular frame and the moving speed of the double knee points are designed as the discriminative features of the worker fall. The proposed method has been verified on the collected dataset and public dataset. The experimental results show that the fall detection accuracy of the proposed method is 95.6% and 96.3%, respectively. Compared with the traditional methods, it has better accuracy and real-time performance and can be applied to worker fall detection in the factory workshop.