Recent advances in training deep learning (DL) models via partial differential equations (PDEs) have gained increasing attention. Such DL techniques are also known as physics-guided deep learning (PGDL). With the aid of the PDEs, PGDL exhibits excellent performance when facing data scarcity. Currently, a great number of PGDL-based frameworks have been proposed for various applications. Despite the tremendous success, PGDL lacks theoretical studies.  In this work, we started by studying the training process of PGDL by the neural tangent kernel (NTK) theory, finding that PGDL tends to be a local approximator after training. Inspired by the observation, we proposed the physics-informed radial basis network that can exhibit local approximating properties throughout the training. It has been proved that the training of PIRBNs through gradient descendant schemes tends to be the Gaussian process regression under appropriate conditions. Moreover, detailed discussions with respect to the initialisation strategies of PIRBN have been presented. Additionally, it is worth highlighting that the extant PGDL techniques, such as adaptive training schemes and domain decomposition, can be applied to PIRBN. In numerical examples, four challenging nonlinear PDEs with high-frequency features and ill-posed computational domains are used to test the performance of the PIRBN. It has been demonstrated that the PIRBN is effective and efficient for those challenging PDEs, while the traditional PGDL can suffer from severe difficulties in minimising training loss.