For midterm electricity load forecasting, modeling methods of feature extraction based on greedy kernel principal component regression(GKPCR) as well as greedy kernel ridge regression(GKRR) are proposed. On the basis of sparse approximation of the kernel matrix, the proposed greedy kernel feature extraction methods aim to find a lower dimensional representation of data embedded in the feature space. Modeling methods of greedy kernel feature extraction have low computational requirements and allow on-line processing of large data sets. The proposed GKPCR and GKRR methods are then applied to electricity peak load forecasting instances in different areas. Compared to existing other kernel-based forecasting methods, experimental results show that, the employed methods may significantly improve the accuracy of peak load forecasting under the same condition, which have considerably better performance, and show the effectiveness and applicability.