In visual object tracking, Transformer-based methods typically fully fine-tune pre-trained models. Full fine-tuning overwrites the general knowledge learned during pre-training, causing catastrophic forgetting and weakening the model’s general visual representations. Moreover, a single network architecture easily suffers from knowledge interference across diverse tracking challenges, limiting generalization. We propose an adaptive sparsity-driven tracker to address these issues. For the encoder, we design a parameter-efficient fine-tuning scheme based on Low-Rank Adaptation (LoRA). It freezes the pre-trained weights to preserve general knowledge and updates only the low-rank adapters for fast domain adaptation. For the decoder, we design a Tri-Regularized Router Mixture of Experts (TR2-MoE) to strengthen expressiveness and scene adaptability. Multiple specialized expert networks handle different tracking challenges, and a dynamic routing mechanism selects scene-adaptive computation paths. Compared with conventional sparse routing, TR2-MoE introduces stochastic regularization during routing and structural constraints inside experts. This design enhances generalization robustness under complex conditions and suppresses excessive co-adaptation among expert units. The load balancing constraint also adopts a normalization strategy aligned with the actual sequence length. This maintains more stable expert utilization during variable-length inference. Extensive experiments on multiple benchmark datasets show that LoMTrack achieves state-of-the-art performance. Notably, it reaches an AUC of 53.8% on the LaSOText dataset, clearly outperforming existing trackers. The experimental results demonstrate stronger dynamic response capability and training robustness across diverse scenarios.