Multi-robot system may be difficult to avoid obstacles and maintain formation in the unknown environment. Based on the improvement of the dynamic window method(DWA), a multi-robot formation control algorithm combining the leader-following method and the behavior-based method is proposed. First, the original DWA is improved by modifying the speed window and three existing evaluation functions and adding two new evaluation functions. As a result, the sample range of speed is increased, the score of the better trajectory is improved, and abilities of navigation to the target and global search in unknown environment are enhanced. Second, based on the formation state and surrounding environment detected in real-time, different behaviors(navigation, obstacle avoidance, tracking and waiting) and the selection method are designed to consider both the obstacle avoidance and maintenance of the formation. Then, the control method of these behaviors is designed based on the improved DWA and social force model(SFM), so that in the unknown environment the leader can plan the path suitable for the whole formation and the follower can adaptively switch the following modes according to the different states of formation. Finally, a series of simulations based on Matlab and V-REP are carried out, whose results show that the improved DWA can significantly improve the traffic efficiency and global search ability of the robot, and the formation control algorithm can achieve the maintenance, obstacle avoidance and transformation of the formation.