Traditional intelligent fault diagnosis methods largely depend on manual feature extraction and expert knowledge. However, complex working conditions of rotatory machinery make traditional fault diagnosis lack adaptivity and generalization. Aiming to solve the problems mentioned above, a novel CNN-based hierarchical fault diagnosis algorithm called CNN-HFD is proposed. Firstly, raw temporal vibration signals are segmented to enlarge samples. Then, several simple CNN networks are constructed according to fault categories and severities. Training samples divided by a certain time-step are sent to the CNN. Finally, signals to be identified are utilized as the input of the CNN-HFD. After hierarchical analysis, the fault category and fault severity are output at the end of the model. Experiments on rolling bearing datasets demonstrate that the proposed method can not only achieve 99.5% fault recognition, but also keep a 97% accuracy under variable loads, which verifies its good robustness and generalization.