To address the problems of large scale variation, strong pose disturbance, obvious local deformation, and insufficient stability of conventional appearance-based matching in global-local dual-field cooperative sorting of slender non-rigid impurities, a lightweight topology-aware and association method for cross-FOV cooperative handover is proposed. First, a cross-FOV topological fingerprint representation mechanism is constructed. Instance segmentation, skeleton extraction, key-node classification, and fingerprint graph construction are used to transform unordered pixel masks into topology graphs with structural semantics, and intrinsic scale normalization is introduced to describe targets uniformly under different fields of view. Second, a lightweight cross-FOV topology matching mechanism guided by geometric priors is presented. Topology-weighted principal-axis normalization is used to reduce rotational disturbance, while structural relation constrained matching and lightweight Siamese learning improve association accuracy and edge-side deployment efficiency under complex conditions. Finally, a global-local cooperative handover strategy based on topology anchors is designed. The topology fingerprint and motion prediction from the global field are used as handover references, and target retrieval, probabilistic discrimination, and handover execution are completed in the local field. Experiments on a cross-FOV tomato sorting system and real scenes show that the proposed method increases the average cross-FOV matching accuracy to 89.2%, achieves a handover success rate of 88.5%, and provides strong robustness against interference.