Passive attitude tracking control without angular velocity measurement is investigated based on the extended state observer(ESO), for the spacecraft attitude motion model described by modified Rodrigues parameters. Firstly, to weaken the peaking phenomenon in the constant gain ESO, a time-varying gain ESO is designed to provide estimations of both angular velocity and total disturbance of the attitude motion system. Then, by considering the passivity property of spacecraft attitude motion, combing the interconnection and damping assignment passivity-based control(IDA-PBC) theory and the backstepping idea, the tracking control law is designed. Finally, the stability of the closed-loop system is proved theoretically. Numerical simulations demonstrate the validity of the proposed control method.