Offline reinforcement learning (ORL) addresses dynamic decision-making problems by utilizing pre-collected datasets, offering significant potential for the application of reinforcement learning in real-world scenarios. Existing ORL methods primarily focus on countering out-of-distribution actions, which increases the likelihood of learning suboptimal policies. To this end, this paper investigates the cognitive error that leads to suboptimality and proposes an uncertainty-aware ORL with mild generalization (UAMG). First, a mild generalization policy consisting of learned policy and perturbation model is designed to adapt to unseen actions. Then, an annealed behavioral cloning penalty is introduced in the policy update network to gradually improve the generalization capability of learned policy. Furthermore, uncertainty is incorporated into the Q-function estimation, and an uncertainty quantifier is constructed using the mild generalization policy, enabling effective quantification of uncertainty and thereby reducing the cognitive error. Theoretical analysis proves that the UAMG can effectively reduce the suboptimality of learned policy. Experimental results on D4RL benchmark demonstrate that compared with comparative methods, the UAMG excels in mitigating the cognitive error and achieving the highest returns in most tasks.