It is commonly believed that the fidelity of quantum teleportation using localized quantum objects with one party or both accelerated in vacuum would be degraded due to the heat up by the Unruh effect. In this paper, we point out that the Unruh effect is not the whole story in accounting for all the relativistic effects in quantum teleportation. First, there could be degradation of fidelity by a common field environment even when both quantum objects are in inertial motion. Second, relativistic effects entering the description of the dynamics, such as frame dependence, time dilation, and Doppler shift, already existent in inertial motion, can compete with or even overwhelm the effect due to uniform acceleration in a quantum field. We show it is not true that larger acceleration of an object would necessarily lead to a faster degradation of fidelity. These claims are based on four cases of quantum teleportation we studied using two Unruh-DeWitt detectors coupled via a common quantum field initially in the Minkowski vacuum. We find the quantum entanglement evaluated around the light cone, rather than the conventional ones evaluated on the Minkowski time slices, is the necessary condition for the averaged fidelity of quantum teleportation beating the classical one. These results are useful as a guide to making judicious choices of states and parameter ranges and estimation of the efficiency of quantum teleportation in relativistic quantum systems under environmental influences.