We show that any Sachdev-Ye-Kitaev- (SYK) like model with finite-body interactions among local degrees of freedom, e.g., bosons or spins, has a fundamental difference from the standard fermionic model: The former model fails to be described by an annealed free energy at low temperature. In this respect, such models more closely resemble spin glasses. We demonstrate this by two means: first, a general theorem proving that the annealed free energy is divergent at low temperature in any model with a tensor product Hilbert space, and second, a replica treatment of two prominent examples which exhibit phase transitions from an "annealed" phase to a "nonannealed" phase as a function of the temperature. We further show that this effect appears only at O(N)th order in a 1/N expansion, even though lower-order terms misleadingly seem to converge. Our results prove that the nonbosonic nature of the particles in the SYK model is an essential ingredient for its physics, highlight connections between local models and spin glasses, and raise important questions as to the role of fermions and/or glassiness in holography.