The consensus problem is investigated for high-order continuous-time linear multi-agent systems with timevarying delay protocol over switching directed communication topologies. By using a linear transformation, the consensus problem is equivalently transformed into the asymptotic stability problem of a corresponding switched delay system. Under the assumption that each of the communication topologies is consensusable, with the help of the scheme of average dwelltime in switched systems, a sufficient condition is presented in terms of linear matrix inequalities(LMIs) to guarantee the consensus of the agents. Simulation result shows the effectiveness of the obtained results.