Optomechanical systems provide a unique platform for observing quantum behavior of macroscopic objects. However, efforts toward realizing nonlinear behavior at the single-photon level have been inhibited by the small size of the radiation pressure interaction. Here we show that it is not necessary to reach the single-photon strong-coupling regime in order to realize significant optomechanical nonlinearities. Instead, nonlinearities at the few quanta level can be achieved, even with weak coupling, in a two-mode optomechanical system driven near instability. In this limit, we establish a new figure of merit for realizing strong nonlinearity which scales with the single-photon optomechanical coupling and the sideband resolution of the mechanical mode with respect to the cavity linewidth. We find that current devices based on optomechanical crystals, thought to be in the weak-coupling regime, may be able to achieve strong quantum nonlinearity, enabling deterministic interactions between single photons.