At ultracold temperatures, the Pauli exclusion principle suppresses collisions between identical fermions. This has motivated the development of atomic clocks with fermionic isotopes. However, by probing an optical clock transition with thousands of lattice-confined, ultracold fermionic strontium atoms, we observed density-dependent collisional frequency shifts. These collision effects were measured systematically and are supported by a theoretical description attributing them to inhomogeneities in the probe excitation process that render the atoms distinguishable. This work also yields insights for zeroing the clock density shift.