This paper presents a linear temporal logic(LTL) guided model-free safe reinforcement learning algorithm to synthesize a control policy that maximizes the satisfaction probability of complex task in an unknown stochastic environment and ensures the safety of agent during learning process. Considering environmental uncertainties, the probabilistic motion of the robot is modeled as a Markov decision process(MDP) with unknown transition probabilities. LTL is applied to describe the complex task, which can be converted to a transition-based limit deterministic generalized Büchi automaton(tLDGBA) with several accepting sets. The accepting frontier function is then designed to record the visited accepting sets, which gives rise to a constrained tLDGBA(ctLDGBA). To ensure the system safety, based on the safety fragment of the LTL formula and the observation function of the MDP, a safety game is constructed to synthesize a shield that ensures the system safety during the learning process. Rigorous analysis shows that the proposed safe reinforcement learning method is guaranteed to obtain the optimal policy that maximizes the probability of satisfying the LTL task while ensuring system safety. The effectiveness of the LTL guided safe reinforcement learning algorithm is demonstrated via simulation results.