Rotating machinery serves as a critical component in industrial systems, and its fault diagnosis is essential for ensuring operational safety. However, deep transfer learning-based fault diagnosis faces two major challenges under varying conditions: the high cost of fault sample collection and significant data distribution discrepancies. These challenges become particularly prominent in open-set scenarios where the target domain contains unknown fault types. To address these issues, this paper proposes a hierarchical clustering guided reliable pseudo-label domain adaptation (HCRPDA) method. First, the feature extractor and classifier are supervised trained using source domain data, while domain confusion loss is constructed to drive adversarial learning between the source and target domains, achieving cross-domain distribution alignment for known classes. Second, hierarchical clustering is performed based on two metrics: the source domain similarity output by the domain discriminator and the classification entropy computed from the classifier’s output, allowing the selection of high-confidence pseudo-labeled samples for unknown classes. These samples are then used to train a dedicated unknown-class discriminator, enhancing the model’s ability to identify unknown faults. Finally, experimental validation on the Paderborn University (PU) bearing dataset and the PHM2009 gearbox dataset demonstrates that the HCRPDA outperforms mainstream domain adaptation approaches in both unknown-class detection rate and known-class classification accuracy, particularly in scenarios where the target domain contains a high proportion of unknown-class samples.