Effectiveness optimization is the important approach to design system-of-system structure and configure scenario schemes. The system-of-system simulation system is usually characterized by complex structure, continuous and discrete mixing, multiple input and output variables, and high operation cost, which leads to some problems of multi-objective, mixed variables, multi-peak and low efficiency, etc. existed in effectiveness optimization. A complex and expensive simulation-based approach to system-of-system effectiveness optimization is proposed. The exploitation- exploration based sequential metamodel strategy is presented coupling clustering with space filling criterion for expensive simulation. The hybrid coding strategy and the corresponding genetic operators are introduced to improve the multiobjective optimization efficiency of the traditional NSGA-II algorithm.The comparative analysis with the state-of-art compensive optimization algorithms shows that optimization results of the proposed algorithm have better convergence, diversity and distribution. Futhermore, the effectiveness simulation optimization for some air defense system-of-system is taken as an example to verify the feasibility and effectiveness of the proposed method.