The graph convolution network has attracted much attention because it can directly process the topological graph of joint points and has good performance in behavior recognition. However, this kind of methods often have the problems of weak long-term information dependence modeling ability and not paying attention to the imbalance between spatial semantics and temporal events. Therefore, a human behavior recognition method based on the time-domain extended residual network and a dual-branch structure is proposed. In the method of spatiotemporal behavior feature extraction, not only the graph convolution is used to extract spatial domain features, but also the extended causal convolution and the residual connection structure are used to construct the time-domain extended residual network to extract time-domain features. The network can effectively expand the receptive field in time domain without increasing a large number of parameters, so as to better obtain the long-term dependence of human joint information in time domain. At the same time, a dual branch structure is constructed, in which the low frame rate branch focuses on extracting rich spatial semantic information with less time frames and more channels, while the high frame rate branch focuses on capturing the rapid changes of human behavior with more time frames and less channels. The accuracy on the NTU RGB $ + $ D data set is higher than the current advanced behavior recognition methods.