As the same object with different spectrum and the different objects with same spectra of high resolution remote sensing image segmentation lead to seqmentation diffculty, a fast remote sensing image segmentation algorithm is proposed by hierarchical Gaussian mixture model (HGMM). Firstly, the HGMM is used to build statistical model of image, which has the ability to model the asymmetrical, heavy-tailed and multimodal distribution of pixel intensities. Then, the HGMM-based segmentation model can be built by following Bayesian theorem. To simplify the complexity of estimating parameters and improve the efficiency, the mean and variance are defined as weights function. Finally, its model parameters can be solved by conjugate gradient method (CGM). The tests can be done with synthetic, panchromatic and color images by using the proposed algorithm and the methods based on traditional statistic model. The results show that the proposed algorithm can obtain accurate segmentation results and high efficiency, which has the ability to model the complicated distribution of pixel intensities.