One manifestation of quantum chaos is a random-matrix-like fine-grained energy spectrum. Prior to the inverse level spacing time, random matrix theory predicts a "ramp" of increasing variance in the connected part of the spectral form factor. However, in realistic quantum chaotic systems, the finite-time dynamics of the spectral form factor is much richer, with the pure random matrix ramp appearing only at sufficiently late time. In this article, we present a hydrodynamic theory of the connected spectral form factor prior to the inverse level spacing time. We first derive a general formula for the spectral form factor of a system with almost-conserved sectors in terms of return probabilities and spectral form factors within each sector. Next we argue that the theory of fluctuating hydrodynamics can be adapted from the usual Schwinger-Keldysh contour to the periodic time setting needed for the spectral form factor, and we show explicitly that the general formulais recovered in the case of energy diffusion. We also initiate a study of interaction effects in this modified hydrodynamic framework and show how the Thouless time defined as the time required for the spectral form factor to approach the pure random matrix result is controlled by the slow hydrodynamics modes. We then extend the formalism to Floquet systems, where a ramp is expected but with a different coefficient, and we derive a crossover formula from the Hamiltonian ramp to the Floquet ramp when the Floquet drive is weak. Taken together, these results establish an effective field theory of chaotic spectral correlations which predicts the random matrix ramp at late time and computes corrections to it at earlier times.