We consider topological superconductors and topological insulator/superconductor structures in the presence of multiple static vortices that host Majorana modes and focus on the Majorana tunneling processes between vortices. It is shown that these tunnelings generally lift the degeneracy of the many-body ground state in a nonuniversal way, sensitive to microscopic details at the smallest length-scales determined by the underlying physical problem. We also discuss an explicit realization of the Jackiw-Rossi zero-mode in a topological insulator/superconductor structure with zero chemical potential. In this case, the exact degeneracy of the many-anyon ground state is protected by an additional chiral symmetry and can be linked to the rigorous index theorem. However, the existence of a nonzero chemical potential, as expected in realistic solid-state structures, breaks chiral symmetry and removes protection, which leads to the degeneracy being lifted. Finally, we discuss the implications of our results for the collective states of many-anyon systems. We argue that quantum dynamics of vortices in realistic systems is generally important and may give rise to effective time-dependent gauge factors that enter interaction terms between Majorana modes in many-anyon systems.