Some background info on my experience, so you can weigh my answer accordingly: I've conducted mate preference studies in various forms on fish, crickets, and wolf spiders (though the latter is the organism in which I have the most experience). In all three study organisms, however, I have always recorded Latency information (e.g., latency to approach, latency to orient, latency to copulate, etc.) and Association Time. Point being, in my opinion, there is never any harm in measuring such data and reporting the results from this information. The difficulty is being careful in how you frame the results. Some things to consider that I would think as a reviewer:

1) In your study, are Latency Time and Total Association time correlated? If so, then both variables might be a reasonable indicator of preference, as long as the statement below is true (see #2). If they are not correlated, but females approach males with Phenotype 1 faster than Phenotype 2 (so, a shorter latency to approach or visit), then that might be more indicative of a sensory bias than a mate preference.

2) Each of your variables (Latency to Approach, Visits, Association Time) are events that lead up to the actual mating event. In our wolf spiders, we can take it a step further and use Receptivity Displays as an indicator of preference because they only use them in the mating context. So yes, a reviewer might decide that your fish might spend more time with one male or another, but it might have nothing to do with an interest in mating with that male. Thus, **if there is a publication in your species that demonstrates that females that spend more time (or approach males more quickly) will also end up mating with these males more frequently, then you could argue that your other behavioral data (latency to approach, association time) would be indicative of mate preferences**. You still might have to be careful though about your phrasing.

3) If the above (see #2) is not available for your specific species, you'll have to cite several publications in other species and hope for the best.

4) If you can't argue (or would prefer to be more conservative) that Latency to Approach indicates "mate preference", it is still an interesting behavioral response. 

5) Don't forget that you cannot "prove" the reliability of the use of the variable. You can only provide additional data that correlates or is associated with such data.

6) See Gabor (1991) "Association patterns of sailfin mollies (Poecilia latipinna): alternative hypotheses". Behavioral Ecology and Sociobiology, 46:333-340 for a good discussion of how association time might not always be indicative of mate preference. I'm sure the citations within it would lead you in the correct direction as well.