Atom interferometers provide a powerful means of realizing quantum coherent systems with increasingly macroscopic extent in space and time. These systems provide an opportunity for a variety of novel tests of fundamental physics, including ultralight dark matter searches and tests of modifications of gravity, using long drop times and microgravity environments. However, as experiments operate with longer periods of free fall and become sensitive to smaller background effects, key questions start to emerge about the fundamental limits to future atom interferometry experiments. We study the effects on atomic coherence from hard-to-screen backgrounds due to baths of ambient particles with long-range forces, such as gravitating baths and charged cosmic rays. Our approach-working in the Heisenberg picture for the atomic motion-makes proper inclusion of the experimental apparatus feasible and clearly shows how to handle long-range forces. We find that these potential backgrounds are likely negligible for the next generation of interferometers, as aggressive estimates for the gravitational decoherence from a background bath of dark matter particles gives a decoherence timescale on the order of years.