This paper addresses the energy-efficient flexible job shop batch scheduling problem (EFJSP-BS) by formulating an optimization model aimed at minimizing both the makespan and the total energy consumption of machines. A rule-based digital batching method is proposed to reasonably divide jobs into batches, further improving scheduling efficiency. To efficiently solve the problem, a nondominated sorting genetic algorithm II integrated with reinforcement learning (RLNSGA-II) is proposed. This algorithm adaptively adjusts the crossover and mutation rates through a reinforcement learning strategy, significantly enhancing global search capabilities. Moreover, three neighborhood search strategies tailored to the characteristics of the batching problem are designed to substantially improve local search performance. Comparative experiments are conducted to validate the effectiveness of the reinforcement learning-based adaptive parameter strategy and the neighborhood search strategies. Additionally, the performance of the RLNSGA-II is compared with other multi-objective optimization algorithms, demonstrating its significant superiority in solving the EFJSP-BS.