Aim to approximately joint diagonalization(JAD) method in solving blind source separation(BSS) problem, a novel non-orthogonal JAD(NJAD) algorithm is proposed. By using the penalty function, the constrained optimization problem of NJAD can be changed to an unconstrained optimization problem. Then the particle swarm algorithm is used to realize the optimization of the criterion. The updated strategy of penalty factor is given and the convergence of the algorithm is analyzed. Computer simulations show the capability of the proposed algorithm in solving the BSS problems compared to another one.