The unresolvable group target tracking algorithm occurs underestimation when the group targets merge, cross and split, and it also occurs excessive partition number and a large amount of calculation. Therefore, this paper proposes a group target Gaussian mixture probability hypothesis density(GMPHD) filtering algorithm based on mean shift(MS) and bilayer group structure(BGS) model. The algorithm uses MS to partition the measurements, at the same time, according to the feedback information of the second layer group structure, it is necessary to judge whether or not to partition the group measurements for second time. Then, the group target GMPHD filtering based on the ellipse random hypersurface model (RHM) is used to predict, update and extract the group target state. Finally, the second group structure is updated by using the extracted state of the group, and the obtained group information is fed back to the measurement partition step. Simulation experiments show that, the proposed algorithm not only has a higher real-time performance, but also solves the underestimation problem of group target number when the group targets merge, cross and split.