Recently the Casimir-Polder force felt by an atom near a substrate under nonequilibrium stationary conditions has been studied theoretically with macroscopic quantum electrodyanamics (MQED) and verified experimentally with cold atoms. We give a quantum field theory derivation of the Langevin equation describing the atom s motion based on the influence functional method valid for fully nonequilibrium (nonstationary) conditions. The noise associated with the quantum field derived from first principles is generally colored and nonlocal, which is at variance with the local source hypothesis of MQED s generalization to nonequilibrium conditions. Precision measurements on the shape deformation of an atomic gas as a function of its distance from a mirror would provide a direct check of our predictions based on the Langevin equation.