This paper is mainly concerned with the optimal control of multiple CSPS system with two-type product mixed flow. How the stations in the system work cooperatively to complete the machining task is one of the important subjects to improve productivity. An operating mode based on equilibrium of products is proposed by using look-ahead range as the control variable of each station and through the local information interaction between stations. A model-free multi-agent reinforcement learning algorithm is used to derive the optimal control policy. Simulation results show the rationality of the proposed operating mode and the effectiveness of the proposed algorithm, and the impact of different parameters on system performance is also analyzed.