Particle-hole symmetry (PHS) of conductance into subgap states in superconductors is a fundamental consequence of a noninteracting mean-field theory of superconductivity. The breaking of this PHS has been attributed to a noninteracting mechanism, i.e., quasiparticle poisoning (QP), a process detrimental to the coherence of superconductor-based qubits. Here we show that the ubiquitous electron-boson interactions in superconductors can also break the PHS of subgap conductances. We study the effect of such couplings on the PHS of subgap conductances in superconductors using both the rate equation and Keldysh formalism, which have different regimes of validity. In both regimes, we found that such couplings give rise to a particle-hole asymmetry in subgap conductances which increases with increasing coupling strength, increasing subgap-state particle-hole content imbalance and decreasing temperature. Our proposed mechanism is general and applies even for experiments where the subgap-conductance PHS breaking cannot be attributed to QP.