We study the intrinsic, disorder-induced decoherence of an isolated quantum system under its own dynamics. Specifically, we investigate the characteristic time scale (i.e., the decoherence time) associated with an interacting many-body system losing the memory of its initial state. To characterize the erasure of the initial state memory, we define a time scale, the intrinsic decoherence time, by thresholding the gradual decay of the disorder-averaged return probability. We demonstrate the system-size independence of the intrinsic decoherence time in different models, and we study its dependence on the disorder strength. We find that the intrinsic decoherence time increases monotonically as the disorder strength increases in accordance with the relaxation of locally measurable quantities. We investigate several interacting spin (e.g., Ising and Heisenberg) and fermion (e.g., Anderson and Aubry-Andre) models to obtain the intrinsic decoherence time as a function of disorder and interaction strength.