Attitude tracking control without angular velocity measurements is investigated by using the passivity of the system and the property of rotation matrix. The control scheme developed on the special orthogonal group(SO(3)) is proposed to avoid ambiguities and singularities of attitude parameters. Firstly, the equations of attitude tracking error are established by using the rotation matrix. Then, the inherent passivity of the system is analyzed to reveal the stability of the closed-loop system. With the use of a novel passive filter, a controller without angular velocity is proposed while the velocity measurements may not be available. The strict Lyapunov stability analysis is also presented. Finally, numerical simulations validate the effectiveness of the proposed control method.