It is difficult for a reinforcement learning agent to learn an optimal policy from scratch, when facing complex tasks with high-dimensional continuous state-space or sparse rewards. How to represent the known knowledge in a form understandable by human beings and the learning agent, and effectively accelerate policy convergence is still a difficult problem. This paper proposes a deep reinforcement learning (DRL) framework integrating with cognitive behavior models. It represents prior knowledge as belief-desire-intention (BDI) based cognitive behavior models, which are used to guide policy learning in DRL. Besides, we introduce the deep Q-learning with the cognitive behavior model (COG-DQN) algorithm and the proximal policy optimization with the cognitive behavior model (COG-PPO) algorithm based on the proposed framework. Moreover, we quantitatively design the guidance strategies of the cognitive behavior model to policy update. Finally, in a typical gym environment and an air combat maneuver confrontation environment, we verify that the proposed algorithms can efficiently use the cognitive behavior model to accelerate policy learning, and significantly alleviate the impact of high-dimensional state-space and sparse rewards.