To deal with the robust model estimation, an improved RANSAC algorithm is proposed based on a total probability updating procedure. In this proposed method, the initial probabilities of the test points are estimated by using a hybrid probability model. Under the framework of the RANSAC algorithm, the evaluations that model estimation and test points fit the consistent set are employed to update the probabilities of test points with the total probability theorem. According to this updated probabilities, an inverse mapping-based sampling method is adopted to improve the convergence rate of the proposed algorithm. Moreover, an analysis procedure is established by using the average probability of the test points, which verifies the convergence of the proposed algorithm. Finally, the simulation and real image matching experiments demonstrate the effectiveness and feasibility of the proposed algorithm.