We establish the universal torus low-energy spectra at the free Dirac fixed point and at the strongly coupled chiral Ising fixed point and their subtle crossover behavior in the Gross-Neuveu-Yukawa field theory with n(D) = 4 component Dirac spinors in D = (2 + 1) dimensions. These fixed points and the field theories are directly relevant for the long-wavelength physics of certain interacting Dirac systems, such as repulsive spinless fermions on the honeycomb lattice or pi-flux square lattice. The torus energy spectrum has been shown previously to serve as a characteristic fingerprint of relativistic fixed points and is a powerful tool to discriminate quantum critical behavior in numerical simulations. Here, we use a combination of exact diagonalization and quantum Monte Carlo simulations of strongly interacting fermionic lattice models, to compute the critical torus energy spectrum on finite-size clusters with periodic boundaries and extrapolate them to the thermodynamic limit. Additionally, we compute the torus energy spectrum analytically using the perturbative expansion in epsilon = 4 - D, which is in good agreement with the numerical results, thereby validating the presence of the chiral Ising fixed point in the lattice models at hand. We show that the strong interaction between the spinor field and the scalar order-parameter field strongly influences the critical torus energy spectrum and we observe prominent multiplicity features related to an emergent symmetry predicted from the quantum field theory. Building on these results we are able to address the subtle crossover physics of the low-energy spectrum flowing from the chiral Ising fixed point to the Dirac fixed point, and analyze earlier flawed attempts to extract Fermi velocity renormalizations from the low-energy spectrum.