In this paper, direct path interference cancellation for a passive radar system is investigated. A new cancellation scheme is proposed to address inevitable hardware discrepancy between the surveillance channel and the reference channel. In the proposed scheme, firstly, the frequency deviation between the two channels is reduced based on the maximum likelihood principle, and the hierarchical search method is used to reduce the complexity, then the distortion in time domain is compensated via a de-convolutional algorithm. The numerical results show that, the proposed scheme can effectively compensate the discrepancy between the two channels and reconstruct the direct interference signals in the surveillance channel accurately, which can improve the interference cancellation performance and is attractive in practical scenarios.