Certain frustrated systems, including spin ice and dimer models, exhibit a Coulomb phase at low temperatures, with power-law correlations and fractionalized monopole excitations. Transitions out of this phase, at which the effective gauge theory becomes confining, provide examples of unconventional criticality. This Letter studies the behavior at nonzero monopole density near such transitions, using scaling theory to arrive at universal expressions for the crossover phenomena. For a particular transition in spin ice, quantitative predictions are made by mapping to the XY model and confirmed using Monte Carlo simulations.