We study the magnetoresistance of two-dimensional bosonic Anderson insulators. We describe the change in spatial decay of localized excitations in response to a magnetic field, which is given by an interference sum over alternative tunneling trajectories. The excitations become more localized with increasing field (in sharp contrast to generic fermionic excitations which get weakly delocalized): the localization length xi(B) is found to change as xi(-1)(B) - xi(-1)(0) similar to B-4/5. The quantum interference problem maps onto the classical statistical mechanics of directed polymers in random media (DPRM). We explain the observed scaling using a simplified droplet model which incorporates the nontrivial DPRM exponents. Our results have implications for a variety of experiments on magnetic-field-tuned superconductor-to-insulator transitions observed in disordered films, granular superconductors, and Josephson junction arrays, as well as for cold atoms in artificial gauge fields.