This paper addresses the issues of severe chattering, slow dynamic response and insufficient noise immunity in voice coil motor servo systems by proposing a fractional-order high-order integral sliding mode control method (AT-FO-FHISMC) based on a hyper-helical sliding mode observer and an adaptive hyperbolic tangent convergence law. By establishing a local model of the voice coil motor, a third-order hyper-helical observer is designed to perform real-time observation and compensation of lumped disturbances in the system, thereby reducing the sliding mode switching gain. Combined with a fractional-order integral sliding mode surface and an improved Oustaloup filter, this approach further suppresses high-frequency chattering and enhances system robustness, and a Lyapunov stability proof is provided. A simulation and hardware-in-the-loop test platform was established. Simulation results show that during step-response tracking, the AT-FO-FHISMC achieves stable performance without overshoot within 47 ms, outperforming the 63 ms of the integral sliding mode control (ISMC) and the 56 ms of the fractional-order fractional-high-order integral sliding mode control (FO-FHISMC). Test results show that the rise time of the AT-FO-FHISMC is 52 ms, which is also significantly better than the 77 ms and 65 ms achieved by the comparative control methods. This method demonstrates excellent overall performance in terms of improving dynamic response speed, suppressing oscillations and enhancing noise immunity, providing a practical solution for high-performance control of voice coil motors and other direct-drive servo systems.