A lightweight image matting framework, which is based on multi-task structure, is designed to meet the requirements of the mainstream computing platforms. Concretely, the overall task can be split into two sub-tasks. One sub-task is to classify the higher-level features at the semantic level, and then it distinguishes foreground/background features from the unknown regions. Another task is to calculate the weights of the linear combination for the foreground and background layers. Accurate foreground features are obtained by sharing the weights of high-level feature networks with feature classification tasks, and they are fused with low-level convolution features. The proposed model outputs more accurate mattes. Also, the convolutional neural network is optimized to lightweight the model. On a benchmark dataset of Composition 1K, schemes performance is compared with different architectures. The proposal can reduce the 19% and 81% of storage consumption in comparison with DIM(deep image matting) and AdaMatting(adaptation and matting) on $640\times640$ images. For the identical data inputs, the running time of the proposed model is only about $1/5$ of DIM's.