Traditional state observers reconstruct the system state only based on the current observation errors, which ignore the system historical observation datas. For the perturbed second-order uncertain linear system, a proportional-integral-retarded sliding mode observer is proposed, which achieves robust and accurate estimation of system states. First, the memory sliding mode function is designed as the linear combination of historical and current observation errors, the design parameters include sliding mode surface gain and artificial time delay. Second, the delayed measurements in the sliding mode surface are expanded based on the Taylor series, and the truncation error is expressed as integral. The memory output feedback equivalent control law is designed based on the delayed dependent Lyapunov functional. On this basis, the dynamic exponential stability analysis and error compensation of sliding mode are performed. Then, the design of observer gains is transformed into a multi-objective optimization problem, with optimization goals: decay rate, control effort, and high-frequency noise insensitivity. Based on the particle swarm algorithm, the optimization of design parameters is realized between the above three optimization goals, and a reasonable compromise among the competitive goals including the rapidness, accuracy and stability. Finally, the feasibility and effectiveness of the proposed sliding mode observer is verified in a passive network system.