To solve the hunting problem of multi-USVs(unmanned surface vehicles) on the sea, a multi-agent reinforcement learning hunting algorithm is proposed. Firstly, the environmental and kinematic model of the boundary-free hunting problem is established based on the background of the cooperative attack of USVs, and the criteria for successful hunting are given according to the requirements of rapidity and encirclement. Then, a Markov decision process framework is established based on the multi-agent PPO(MAPPO) algorithm. The state-space with scalability and permutation invariant, an action space with decoupling of capture distance and azimuth, and a reward function combining capture reward and step reward are designed. Finally, the framework of centralized training and distributed execution is adopted to train the policy. During the training, the skills of curriculum learning are used to make the network converge quickly, and the USVs share the same strategy and execute the action independently. Simulation shows that the proposed method has more advantages than other algorithms in the hunting success rate and timeliness under different testing conditions. In addition, when some of the USVs are failed, the remaining USVs can continue the task, which proves strong robustness and potential for deployment in a real environment.