The leader-following consensus problem of multi-agent systems with general linear systems is studied. It is assumed that each agent has only access to the relative output measurements with respect to its neighbors. Two cases are considered: the directed network with fixed topology and the undirected networks with switching topologies. For both of cases, a distributed observer-type consensus protocol only based on output measurements is proposed. It is proved that the leader-following consensus can be achieved only assuming stabilizability and detectability of the individual systems and imposing very mild connectedness assumption on the interconnection structure by using Lyapunov theory. Finally, simulation results show the effectiveness of the theoretical results.