An improved Mask Region-Convolutional Neural Network (Mask R-CNN) algorithm is proposed to conquer the shortcoming such as the slow convergence rate, low accuracy in the original Mask R-CNN algorithm to obtain contours of fascicular groups from MicroCT images of peripheral nerve. First, the dataset of images was constructed and divided into two subsets. Second, the network architecture with dense connection is proposed to abstract the feature of fascicular groups. Third, the regulation of proposal box scores in object detection part was improved. Additionally, the transfer learning strategy was combined with Mask R-CNN in training process. The average precision (AP) and the Intersection over Union (IoU) are adopted as evaluation indices of algorithm accurate, and the precision threshold is adopted as the evaluation index of algorithm precision. It was the first time identifies the best values of the precision threshold. Experiment results show that the AP and the IoU of the improved approach exceeds 83% and 87% in two peripheral nerve MicroCT image subsets. The improved algorithm has the best contours of fascicular groups at the threshold of 0.85. Experiments show that the improved algorithm can extract the contours of fascicular groups exactly and lay the foundation for the three dimensional visualization of the internal structure of peripheral nerve.