We analyze the fidelity of a deterministic quantum phase gate (QPG) for two photons counterpropagating as polaritons through a cloud of Rydberg atoms under the condition of electromagnetically induced transparency (EIT). We provide analytical results for the phase shift of the quantum gate, and provide an estimation for all processes leading to a reduction to the gate fidelity. Especially, the influence of losses from the intermediate level, dispersion of the photon wave packet, scattering into additional polariton channels, finite lifetime of the Rydberg state, as well as effects of transverse size of the wave packets are accounted for. We show that the flatness of the effective interaction, caused by the blockade phenomena, suppresses the corrections due to the finite transversal size. This is a strength of Rydberg-EIT setup compared to other approaches. Finally, we provide the experimental requirements for the realization of a high fidelity QPG using Rydberg polaritons.