This paper aims to design an optimal passive fractional-order proportional-integral-derivative(PFoPID) controller for grid connected photovoltaic inverters(PV), which can achieve maximum power point tracking(MPPT) under different atmospheric conditions via the perturb and observe(P&O) technique. Firstly, a storage function is constructed based on the tracking errors, in which the beneficial terms are retained to increase the tracking rate. Meanwhile, other system nonlinearities are fully compensated to realize globally consistent control performance. Then, the fractional-order PID(FoPID) control framework is introduced as the additional input to reshape the storage function, and optimal control parameters are tuned by using the grouped grey wolf optimizer(GGWO). Three case studies are carried out, e.g., solar irradiation variation, temperature variation, and power grid voltage drop. Simulation results verify that the PFoPID control outperforms the conventional PID control, FoPID control, and passive-based control(PBC) under different operation conditions. Finally, a hardware-in-loop(HIL) test based on dSpace is undertaken to validate the implementation feasibility of the proposed approach.