In the paper, we propose a physics-informed recurrent neural network (PIRNN-FDM) for time-dependent partial differential equations, which has a network structure similar to FDM. In our scheme, LSTM cells are adopted to ensure the continuity of field variables in time stepping. In order to preferably accelerate the learning process, the predicted values of the current layer are used as the input of the next layer, which is analogous to FDM. Thus, the field values at different time step can be obtained. Besides, the physical information is also fully utilized, in which the residual of the governing PDE and the loss of the initial conditions and boundary conditions are used to construct the loss function. Finally, we conduct intensive time-dependent PDEs to demonstrate the performance of PIRNN-FDM. Through the results, it is found that PIRNN-FDM can accurately predict the field values at different time even if a relatively large time step is taken, which is not constrained by the proportion of space and time.