Facial expression recognition is still faced with great challenges due to the intra class variation and interclass interference. In this paper, a method of deep facial expression recognition based on random forest with gender constraints is proposed to solve the problems of noise, gender and other variation and interference. Firstly, robust facial features are extracted by deep multi-instance learning to solve the problems of illumination, occlusion and low resolution. Secondly, the face expression classifier is designed by using gender conditional random forest classification to solve the problem of gender interference. Extensive experiments on the public CK+, BU-3DEF and LFW databases show that the recognition rate of the proposed method is 98.83%, 90% and 60.58% respectively, which has better performance and robustness compared with the state-of-the-art methods. In addition, compared with other advanced deep learning methods (requiring a large number of training databases), the proposed method only needs a small number of training samples to achieve better results.