Under the condition that the multimodal data of image and text are available, the use of one deep network to make fusion decisions is a usual way. However, it faces the challenges of interpretability and reliability. To address these two challenges, this article proposes a two-stage interval decision-making method driven by image and text data, including the criterion prediction and criterion aggregation stages. In the criterion prediction stage, a set of deep networks is determined according to the requirements of extracting image characteristics, and then a statistical comparison method of deep network performance is designed based on sampling with replacement to determine the rankings of the deep networks in the set. With the resulting rankings, a method of selecting the sequential combinations of deep networks is constructed to identify the reliable optimal combination, which can be used to derive interval-valued predictions from images. In the criterion aggregation stage, criterion weights are learned from the training dataset of text, and then an optimization is constructed based on the verifying dataset of text to learn adaptive weight function. With the resulting criterion weights and weight function, the interval-valued predictions derived from images on each criterion are finally combined to generate the explainable overall predictions. Based on the image and text data collected from the ultrasonic department of a tertiary hospital in Hefei, Anhui, the proposed method is used to help diagnose breast lesions, which validates its effectiveness.