An improved online adaptive modular neural network is proposed for the online modular neural network which is difficult to effectively divide sample space in real time. Firstly, the proposed network performs on-line task decomposition by real-time updating local sample density and RBF neuron center based on distance and density. Then the number of sub-network modules is adaptively increased or decreased according to the number of divided sample spaces, and each sub-network module learns the corresponding sample. Finally, integration module integrates the output of sub-network modules and outputs the final result. For the problem of the online gradient descent method requiring the sample with enough randomness, an online gradient descent method with fixed memory is proposed. Based on the prediction of the typical nonlinear time varying systems and the effluent ammonia nitrogen values from waste water treatment processes, the experimental results show that, the proposed network can effectively update the RBF neuron center in real time, which reduces unnecessary increase or decrease of sub-network modules during the learning process, has more concise structure and can accurately predict the time-varying systems.