The manifold learning methods of the simple graph model ignored the high-order relationship between data points. Therefore, an algorithm, called hyper-graph regularized concept factorization(HRCF) is proposed. HRCF considers the high-order relationship of samples by constructing the hyper-edge in hyper-graph with a subset of data points sharing with some attribute. The concept factorization(CF) algorithm can preserve the high-order relationship of the manifold structure, by adding hyper-graph regulation term in clustering. Thus, the algorithm has more discrimination power. The experimental results on Yale, USPS and TDT2 database show that the proposed approach provides a better representation and achieves better clustering results in terms of accuracy and normalized mutual information, and verify the effectiveness of the proposed method.