Object detection is an important research in the field of computer vision. Traditional target detection methods spend a lot of time on feature extraction, and manual features are not robust to the problem of diverse targets. Deep learning technology has gradually become a breakthrough in computer vision in recent years. By using the classical convolutional neural network VGGNet as a basic network, a network framework for target detection is built by adding some deep networks and combining with the SSD(single shot multi-box detector) algorithm, and an algorithm of feature fusion based SSD(FF-SSD) is proposed. Aiming at the problem of sample imbalance during the training of the model, the original loss function is modified according to the principle of hard example mining. The background is regarded as a simple sample and a modulation factor is introduced to reduce the proportion of background loss to the confidence loss, which makes the model be trained more fully and converge faster, and the target detection accuracy under the complex background is promoted as a result. Meanwhile, for poor detection effect to small targets of the SSD algorithm, the feature pyramid is constructed according to the feature maps extracted from each convolutional layer. Appropriate feature maps are selected and fused to form a new feature map for the prediction. The semantic information fusion is strengthened to enhance the detection accuracy of small targets in order to improve the overall detection accuracy. Experimental results show that the proposed target detection algorithm can achieve high detection accuracy for all kinds of targets in the complex background.