Certain patterns of symmetry fractionalization in topologically ordered phases of matter are anomalous, in the sense that they can only occur at the surface of a higher dimensional symmetry-protected topological (SPT) state. An important question is to determine how to compute this anomaly, which means determining which SPT hosts a given symmetry-enriched topological order at its surface. While special cases are known, a general method to compute the anomaly has so far been lacking. In this paper we propose a general method to compute relative anomalies between different symmetry fractionalization classes of a given (2+1)D topological order. This method applies to all types of symmetry actions, including anyon-permuting symmetries and general space-time reflection symmetries. We demonstrate compatibility of the relative anomaly formula with previous results for diagnosing anomalies for Z(2)(T) space-time reflection symmetry (e.g. where time-reversal squares to the identity) and mixed anomalies for U(1) x Z(2)(T) and U (1) (sic) Z(2)(T) symmetries. We also study a number of additional examples, including cases where space-time reflection symmetries are intertwined in non-trivial ways with unitary symmetries, such as Z(4)(T) and mixed anomalies for Z(2) x Z(2)(T) symmetry, and unitary Z(2) x Z(2) symmetry with non-trivial anyon permutations.