In order to improve the environment cognition ability of intelligent vehicles, this paper proposes a hierarchical factor graph based environment perception and situation cognition model according to the abstraction level of data information. Firstly, based on the hierarchical memory mechanism of human driving cognition, according to the abstract level of processed information from low to high, environment cognition is divided into two task modules: object perception and situation awareness. Then, the topological structure of the hierarchical factor graph is determined and the hierarchical factor graph model is realized. The object perception layer is embodied in multi-source information fusion and object tracking. The situation awareness layer is embodied in situation prediction such as vehicle lane change. Finally, based on PreScan simulation environment data, NGSIM real driving data set and DBNet real data set, the effectiveness of the proposed method is verified. Compared with the existing Kalman filter method and hidden Markov model method, the advantages of hierarchical factor diagram in tracking, fusion, situation prediction accuracy and accuracy are verified.