We study mode competition in a multimode "phonon laser" comprised of an optical cavity employing a highly reflective membrane as the output coupler. Mechanical gain is provided by the intracavity radiation pressure, to which many mechanical modes are coupled. We calculate the gain and find that strong oscillation in one mode suppresses the gain in other modes. For sufficiently strong oscillation, the gain of the other modes actually switches sign and becomes damping, a process we call "anomalous cooling." We demonstrate that mode competition leads to single-mode operation and find excellent agreement with our theory, including anomalous cooling.