We study symmetry-enriched topological (SET) phases in 2+1 space-time dimensions with spatial reflection and/or time-reversal symmetries. We provide a systematic construction of a wide class of reflection and time-reversal SET phases in terms of a topological path integral defined on general space-time manifolds. An important distinguishing feature of different topological phases with reflection and/or time-reversal symmetry is the value of the path integral on non-orientable space-time manifolds. We derive a simple general formula for the path integral on the manifold Sigma(2) x S-1, where Sigma(2) is a two-dimensional non-orientable surface and S-1 is a circle. This also gives an expression for the ground state degeneracy of the SET on the surface Sigma(2) that depends on the reflection symmetry fractionalization class, generalizing the Verlinde formula for ground state degeneracy on orientable surfaces. Consistency of the action of the mapping class group on non-orientable manifolds leads us to a constraint that can detect when a time-reversal or reflection SET phase is anomalous in (2+1)D and, thus, can only exist at the surface of a (3+1)D symmetry protected topological (SPT) state. Given a (2+1)D reflection and/or time-reversal SET phase, we further derive a general formula that determines which (3+1)D reflection and/or time-reversal SPT phase hosts the (2+1)D SET phase as its surface termination. A number of explicit examples are studied in detail.