The time-dependent vehicle routing problem with time windows (TDVRPTW) of fresh e-commerce distribution is studied by considering both economic cost and environmental cost. A calculation method for road travel time across time periods is designed. A freshness measure function of agricultural products and a measure function of carbon emission rate are employed by considering time-varying vehicle speeds, fuel consumptions, carbon emissions, perishable agricultural products, customers'time windows and minimum freshness. A TDVRPTW model is formulated with minimum freshness constraint. The object of the TDVRPTW model is to minimize the sum of economic cost and environmental cost. According to the characteristics of the model, an adaptive improved ant colony algorithm is designed. Finally, the experimental data show that the proposed approaches effectively avoid traffic congestions, reduce total distribution costs, and promote energy conservation and emission reduction.