Multi-object tracking is a crucial technique of video surveillance. Over the past decade, the convolutional neural networks(CNNs) and especially graph neural networks(GNNs) have made multi-object tracking a great progress, where the GNN show an significant advantages due to modeling the relationship between targets and trajectories. These GNN models, however, mostly consider building a global relationship model for targets and trajectories only in two neighboring frames, neglecting the interactions between an object with the others within a frame. In order to handle this issue, we propose an intra-frame relationship modeling and self-attention fusion method for multi-object tracking. Within a frame, the INAF-GNN builds a relational graph model for an object and its neighboring objects to obtain local tracking features. Across two frames, the INAF-GNN constructs another relational graph model for objects and trajectories to acquire global tracking features. In further, both the local and global tracking features are fed into a feature integration module via a self-attention mechanism. We run various experiments on the pedestrian MotChallenge benchmark datasets, and the experimental results show that the proposed method outperforms GNN-based multi-object tracking methods by 1.9% of MOTA and 3.6% of IDF1. Besides, it is also validated over the vehicle UA-DETRAC datasets. Both demonstrate the effectiveness and generalization capability of the proposed method.