We numerically study the effect of short-ranged potential disorder on massless noninteracting three-dimensional Dirac and Weyl fermions, with a focus on the question of the proposed (and extensively theoretically studied) quantum critical point separating semimetal and diffusive-metal phases. We determine the properties of the eigenstates of the disordered Dirac Hamiltonian (H) and exactly calculate the density of states (DOS) near zero energy, using a combination of Lanczos on H-2 and the kernel polynomial method on H. We establish the existence of two distinct types of low-energy eigenstates contributing to the disordered density of states in the weak-disorder semimetal regime. These are (i) typical eigenstates that are well described by linearly dispersing perturbatively dressed Dirac states and (ii) nonperturbative rare eigenstates that are weakly dispersive and quasilocalized in the real-space regions with the largest (and rarest) local random potential. Using twisted boundary conditions, we are able to systematically find and study these two (essentially independent) types of eigenstates. We find that the Dirac states contribute low-energy peaks in the finite-size DOS that arise from the clean eigenstates which shift and broaden in the presence of disorder. On the other hand, we establish that the rare quasilocalized eigenstates contribute a nonzero background DOS which is only weakly energy dependent near zero energy and is exponentially small at weak disorder. We also find that the expected semimetal to diffusive-metal quantum critical point is converted to an avoided quantum criticality that is "rounded out" by nonperturbative effects, with no signs of any singular behavior in the DOS at the energy of the clean Dirac point. However, the crossover effects of the avoided (or hidden) criticality manifest themselves in a so-called quantum critical fan region away from the Dirac energy. We discuss the implications of our results for disordered Dirac and Weyl semimetals, and reconcile the large body of existing numerical work showing quantum criticality with the existence of these nonperturbative effects.