Due to the limited visibility of unmanned surface vessels (USVs) in multi-USV formation systems, this paper proposes a model with an unmanned aerial vehicles (UAV) as the leader of the USV formations. In this paper, a fixed-time optimal formation control algorithm is designed for the formation of multi-USVs under UAV leadership. Building upon the three-degree-of-freedom dynamic modeling of USVs and UAVs, an augmented system is constructed. A performance index function that considers input cost and formation error is established. In the proposed reinforcement learning algorithm with an Identifier-Actor-Critic structure, unknown nonlinear system dynamic is approximated by the identifier. The actor updates through value evaluations of the optimal performance index function. Iterative control policy adjustments conducted by the critic, the control strategy is continuously optimized towards the optimal solution. Furthermore, it is proven that the formation error of the UAV-USV system can converge to a small region around zero within a fixed time. The effectiveness of the proposed algorithm is verified by simulation examples.