The P100 and the N220 are very different beasts, but both are only loosely related to the signal travel time from retina to cortex. There is no direct correlate for that time, as seems to be the assumption behind the question. I am more familiar with the P100, so, I’ll answer that. Its main clinical relevance stems from the fact that (1) its latency is very reliably a certain value in the adult, healthy human, namely something like 104 ms ± 7 ms SD if I remember correctly, and (2) that when the myelin sheaths of the optic nerve are somehow affected, as often happens in multiple sclerosis, that latency is significantly prolonged.
I think that the P100 is taken into consideration because it is the most repeatable wave, both in the same subject and between different subjects, in intraday and witinday re-tests. Regarding the retino-cortical time, perg is needed in the same session.