We give a first principles derivation of a master equation for the evolution of a quantum matter field in a linearly perturbed Minkowski spacetime, based solely on quantum field theory and general relativity. We make no additional assumptions nor introduce extra ingredients, as is often done in alternative quantum theories. When the quantum matter field is projected to a one-particle state, the master equation for a non-relativistic quantum particle in a weak gravitational field predicts decoherence in the energy basis, in contrast to most existing theories of gravitational decoherence. We point out the gauge nature of time and space reparameterizations in matter-gravity couplings, and warn that intrinsic decoherence or alternative quantum theories invoking stochastic dynamics arising from temporal or spatial fluctuations violate this fundamental symmetry of classical general relativity. Interestingly we find that the decoherence rate depends on extra parameters other than the Planck scale, an important feature of gravitational decoherence. This is similar to the dependence of the decoherence rate of a quantum Brownian particle to the temperature and spectral density of the environment it interacts with. The corresponding features when gravity acts as an environment in decohering quantum objects are what we call the textures of spacetime. We point out the marked difference between the case when gravity is represented as a background spacetime versus the case when gravity acts like a thermodynamic bath to quantum particles. This points to the possibility of using gravitational decoherence measurements to discern whether gravity is intrinsically elemental or emergent.