In the real world, the contact operation tasks need to implement the contact force control at the same time implementation of the position control. In order to solve this problem, a hybrid control scheme based on vision and force is proposed. First, a method to estimate the normal vector of the unknown constraint surface is introduced, on-line force feedback data are employed to estimate the local shape of constraint. Second, the image Jacobian is estimated using the recursive least squares algorithm. Finally, an adaptive hybrid visual servoing/force controller is proposed, the direction of the force control becomes perpendicular to the visual servoing control, variable structure controller is applied for visual servoing, PI controller control is applied for force control. Experimental results are shown to demonstrate the effectiveness of the proposed scheme.