We demonstrate fast nonlinear optical switching between two laser pulses with as few as 140 photons of pulse energy by utilizing strong coupling between a single quantum dot (QD) and a photonic crystal cavity. The cavity-QD coupling is modified by a detuned pump pulse, resulting in a modulation of the scattered and transmitted amplitude of a time synchronized probe pulse that is resonant with the QD. The temporal switching response is measured to be as fast as 120 ps, demonstrating the ability to perform optical switching on picosecond timescales.