The paper takes into account the impact between the gate system and the yard system to make full use of terminal resources, reduces the trucks turnaround time, and alleviates the congestion at the terminals. This paper optimizes the truck task assignment and sequence through evaluating the collection and distribution operations as a two-stage hybrid flow shop. Specifically, trucks are considered workpieces that must go through gate operations in the first stage and yard operations in the second stage to complete pickup/delivery operations. The optimization model for collection and distribution operations based on the hybrid flow shop is established, taking into account the realistic constraints such as the non-crossing and safety spacing between the yard cranes. To solve this model, a variable neighbourhood search algorithm based on $Q\text{-}{\rm learning} $ is developed, and search operators are designed based on the problem characteristics. Numerical experiments show the effectiveness of the proposed algorithm. In comparison to conventional strategies, the hybrid flow shop-based strategy reduces the maximum flow time of trucks by 3.70% and reduces the imbalance of operating time between equipment by 17.78%.