Pickup and delivery problems in multi-floor environments pose significant challenges for automated transportation systems, mainly due to the sparse inter-floor connectivity caused by the limited number of elevators. This results in loosely connected subregions, which makes it difficult for existing path optimization methods to generate high-quality cross-floor routes. Although deep reinforcement learning (DRL) has shown strong potential in vehicle routing problems, most DRL-based approaches struggle to effectively model such sparse cross-floor topologies. To address this issue, this paper proposes a DRL-based sparse cross-floor path optimization (DRL-SCFO) method. In DRL-SCFO, the encoder jointly extracts spatial features within individual floors and across different floors, and integrates them through an adaptive fusion strategy to capture complex cross-floor topological structures. The decoder decomposes each decision step into two stages: a global planning stage to determine a potential target, followed by a local execution stage to select a feasible path node guided by this target. Experimental results demonstrate that DRL-SCFO achieves superior solution quality compared with existing DRL-based routing optimization methods on multi-floor pickup and delivery problems, and shows good adaptability across different task topologies and problem scales.