Graphene plasmon has been attracting interests from both theoretical and experimental research due to its gate tunability and potential applications in the terahertz frequency range. Here, we propose an effective scheme to unidirectionally excite the graphene plasmon by exploiting magneto-optical materials in the famous attenuated total reflection (ATR) configuration. We show that the graphene plasmon dispersion relation in such a device is asymmetric in different exciting directions, thus making it possible to couple the incident light unidirectionally to the propagating plasmon. The split of absorption spectrum of graphene clearly indicates that under a magnetic field for one single frequency, graphene plasmon can only be excited in one direction. The possible gate tunablity of excitation direction and the further application of the proposed scheme, such as optical isolator, also are discussed.