When an unmanned surface vehicle (USV) navigates autonomously in congested waters, a large number of other vessels and static obstacles, ocean environment disturbances, and USV uncertain dynamics exacerbate USV collision risks (CRs). Therefore, a USV intelligent collision avoidance (COLAV) decision-making and trajectory tracking control scheme is designed. Firstly, the congestion degree are assessed according to the numbers of other vessels and static obstacles in USV sailing waters, which are introduced into an existing CR index assessment function, such that the CRs of the USV in congested waters are assessed reasonably. Then, a bidirectional long short-term memory neural network (NN) is established to estimate the USV trajectory prediction errors in a future finite time domain based on the USV motion mathematical model nominal parameters, according to input sequences consisting of the USV trajectory tracking control signals and the passed trajectory in a past finite time domain. The NN's weights are updated online, such that the estimated USV trajectory prediction errors can be used to correct the USV trajectory prediction errors caused by ocean environment disturbances and USV uncertain dynamics, implementing the USV intelligent trajectory prediction with self-learning ability. Finally, based on the idea of model predictive control, we obtain the desired trajectory and control signals for USV COLAV maneuvers by solving the optimization problem whose objective is to minimize the USV remaining sailing distance and CR, constrained by the USV dynamics and the safe distances from the USV to other vessels and static obstacles. Simulation and simulation comparison results show that under the proposed intelligent COLAV decision-making and trajectory tracking control scheme, the USV actual sailing distance of its autonomous navigation is shorter and the CR of the USV is lower.