When dealing with adaptive optimal output regulation problems of linear continuous-time systems, the development of an optimal feedback-feedforward controller usually relies on the solution to regulator equations. However, it is nontrivial to obtain this solution when the system dynamic is uncertain. Existing studies of adaptive optimal output regulation often begin with selecting basis of solution to regulator equations and then use online data to learn the weights. But this method is computationally expensive if the order of the plant or the exosystem is high. In this paper, we propose a novel model-free solution to adaptive optimal output regulation problems where the solution to regulator equations can be approximated via gradient descent methods. Notably, the gradient can be learned by online data instead of using model knowledge. Simulation results demonstrate the effectiveness of the proposed method.