The broad learning system (BLS) is an accurate and efficient machine learning algorithm proposed recently, which has shown excellent performance in classification, regression and other problems. However, the traditional BLS takes least squares as learning criterion, which is prone to be affected by outliers and thus generates inaccurate learning models. To solve this problem, this paper proposes a robust broad learning system (RBLS) based on an M-estimator. Different from the BLS, the RBLS uses a robust M-estimator cost function to replace the traditional least squares cost function in the learning model, and adopts the Lagrange multiplier method and the iteratively reweighted least squares method to seek for an optimal solution. In the iterative learning process, the normal sample and the outlier sample will be reversely assigned different weights according to the size of their training errors, so as to effectively suppress or eliminate the adverse effects of the outlier residual on the learning model. Experimental results show that, as a unified robust learning framework, the RBLS can combine different M-estimator weighting strategies and achieve better generalization performance and robustness than the comparison algorithms.