The quadtree is a hierarchical-type data structure, which is a simple, yet efficient technique for nonuniform and adaptive mesh refinement. This study presents a novel, and stable scaled boundary finite element formulation to model the structural response of elasto-plastic material using quadtree meshes. In elasto-plastic analysis, the use of one-point integration is especially attractive due to its computational efficiency. This is achieved by reducing number of locations required to evaluate the constitutive equations in each element. In this formulation, the elasto-plastic material constitutive matrix is assumed to be constant in each quadtree cell once a cell reaches a plastic state. The standard scaled boundary finite element stiffness matrix is then decomposed into two parts with respect to the strain gradient matrix. The first part is the under-integrated stiffness matrix, which is similar to the one-point integration stiffness in standard finite element method. This assumption implies that the stresses are constant within a cell when it is yielding. The second part of the stiffness matrix is treated as a stabilization stiffness matrix to overcome spurious modes and ensures that the formulation is stable. The internal force vector can be evaluated accordingly with respect to these two parts. The efficiency and accuracy of this approach is demonstrated using numerical benchmark examples.