In recent years, with the continuous development of deep learning technology, various semantic segmentation algorithms based on deep learning have emerged, but most of the segmentation algorithms cannot achieve high speed and high accuracy at the same time, and a real-time semantic segmentation framework for multi-channel depth-weighted aggregation networks (MCDWA_Net) is proposed to solve this problem. Firstly, the multi-channel idea is introduced to construct a three-channel semantic representation model, which is used to extract three types of complementary semantic information of the image: 1) Low-level semantic channel outputs the local features such as the edge, color, and structure of the object in the image; 2) Auxiliary semantic channel extracts the transition information between low-level semantics and high-level semantics, and realizes multi-layer feedback to the high-level semantic channel; 3) Advanced semantic channel obtains context logical relationships and category semantic information in images. Then, a three-class semantic feature weighted aggregation module is designed to output a more complete global semantic description. Finally, an enhancement training mechanism is introduced to realize the feature enhancement in the training stage, thereby improving the training speed. Experimental results show that the proposed method not only has fast inference speed, but also has high segmentation accuracy in complex scenes, which can achieve the balance of semantic segmentation speed and accuracy.