In order to solve finite horizon optimal control problems, the adaptive dynamic programming(ADP) algorithm demands the system can reach zero in one step of control. For the nonlinear systems which cannot be controlled to zero in one step, an improved ADP algorithm is presented, and the initial cost is constructed by arbitrary finite horizon admissible sequence. After giving the iterative process, the convergence analysis of the improved algorithm is conducted. If the approximation error of the critic network is considered and several assumptions are satisfied, the iterative cost function will converge to a finite neighborhood of the optimal cost function. A simulation example is provided to verify the effectiveness of the presented approach.