In recent years, value function estimation bias correction has become an important research direction in the field of deep reinforcement learning. Most existing research work focuses on how to alleviate overestimation bias, but ignores the problem of underestimation bias introduced in the process of mitigating overestimation bias. To this end, this paper flexibly sets up multiple Actor and Critic networks in the Actor-Critic framework to alleviate the value function underestimation bias, and proposes a delayed depth deterministic policy gradient based on combined network optimization (D3PG-CNO). The main idea of the D3PG-CNO is to use a Critic network to evaluate the output actions of multiple Actor networks in the experience collection phase, and to select the optimal actions to store in the experience pool. In the experience training stage, the Critic network with the smallest estimated result under the current state-action pair is selected from multiple Critic networks and used to evaluate the output actions of multiple Actor networks, and the maximum evaluation value is selected to calculate the target value. Experimental results on the MuJoCo platform show that the D3PG-CNO significantly reduces estimation bias compared to existing deterministic policy gradient algorithms, improves the stability and convergence speed of the algorithm, and shows better performance in multiple tasks.