When a target object with random orientation grasped by a dual-arm system carries out a trajectory tracking task under the sight of a fixed camera, not only the uncertainties in kinematics and dynamics generated from random orientation of the target object should be taken into consideration, but also the cooperation in kinematics and dynamics does. To address such problems, the adaptive method is adopted to estimate the position information of the mass center and feature points, the fuzzy logic system is used to approximate the unknown dynamics model, and the decentralized method is utilized to cope with the cooperation problem for dual-arm. The image based fuzzy adaptive trajectory tracking controller for the dual-arm system is designed on the basis of the position/force hybrid control method, and the system stability is rigorously proved by using Lyapunov method. Finally, the effectiveness of the proposed controller is demonstrated in the simulation of trajectory tracking.