The information entropy, proposed by Shannon, has been widely used in rough sets. In this paper, we use the rough entropy in rough sets to detect outliers, and propose a rough entropy-based outlier detection approach, which is applied to unsupervised intrusion detection. Firstly, we propose a new definition for outliers based on rough entropy, and design an algorithm called rough entropy-based outlier detection(REOD) to find such outliers. Then, we regard intrusion activities as outliers and apply the REOD to intrusion detection, from which a novel approach for unsupervised intrusion detection is obtained. Experiments on several data sets demonstrate that the REOD performs well for outlier detection. In addition, compared with existing intrusion detection methods, the REOD can detect attacks with high detection rate and low false positive rate. Especially, the computational cost of the REOD is low, and it is suitable for intrusion detection on massive and high dimensional data.