At low numbers, it doesnt matter. If you exqgerate the numbers the effect is more clear.
Eg. if the latency was 100ms, it would feel your movments are behind by 100ms, which would be unplayable.
But if you had a typical latency of 10ms, with rare spikes to 1s, the spikes would be considered lag, and annoying, but most of the time its good and playable.
When I last looked into it, many years ago, RT definitely did negatively impact average latency. It was slower, but consistent. Has that actually changed?