Quantum speed limits such as the Mandelstam-Tamm or Margolus-Levitin bounds offer a quantitative formulation of the energy-time uncertainty principle that constrains dynamics over short times. We show that the spectral form factor, a central quantity in quantum chaos, sets a universal state-independent bound on the quantum dynamics of a complete set of initial states over arbitrarily long times, which is tighter than the corresponding state-independent bounds set by known speed limits. This bound further generalizes naturally to the real-time dynamics of time-dependent or dissipative systems where no energy spectrum exists. We use this result to constrain the scrambling of information in interacting many-body systems. For Hamiltonian systems, we show that the fundamental question of the fastest possible scrambling time—without any restrictions on the structure of interactions—maps to a purely mathematical property of the density of states involving the non-negativity of Fourier transforms. We illustrate these bounds in the Sachdev-Ye-Kitaev model, where we show that despite its “maximally chaotic” nature, the sustained scrambling of sufficiently large fermion subsystems via entanglement generation requires an exponentially long time in the subsystem size.