Electron-phonon coupling is known to play an important role in the charge dynamics of semiconductor quantum dots. Here we explore its role in the combined charge-photon dynamics of cavity-coupled double quantum dots. Previous work on these systems has shown that strong electron-phonon coupling leads to a large contribution to photoemission and gain from phonon-assisted emission and absorption processes. We compare the effects of this phonon sideband in three commonly investigated gate-defined quantum dot material systems: InAs nanowires and GaAs and Si two-dimensional electron gases (2DEGs). We compare our theory with existing experimental data from cavity-coupled InAs nanowire and GaAs 2DEG double quantum dots and find quantitative agreement only when the phonon sideband and photoemission processes during lead tunneling are taken into account. Finally, we show that the phonon sideband also leads to a sizable renormalization of the cavity frequency, which allows for direct spectroscopic probes of the electron-phonon coupling in these systems.