Native speakers of any language have an innate ability to recognize the sounds of their language according to cognitive phonology. Every native speaker has recognition memory and declarative memory which they use to tell what sounds are distinctive in their language - both vowels and consonants.

I think until now science has no answer to your question.

The concept of phonemes: It is an artificial concept. Nobody knows whether there exists such a concept in our minds.

You might want to look into literature on speaker nomalization (e.g. Johnson: The role of perceived speaker identity in F0 normalization of vowels, J. Acoust. Soc. Am. 88, 642 (1990); http://dx.doi.org/10.1121/1.399767) for vowel perception as a starting point.

Best Xaver

Formants are determined by the vocal tract area function and length, which varies with speakers and vowels. The variance in formants between speakers can be found through speech signal analysis. Speaker normalization to reduce the variance in formants caused by voval tract length is needed in automatic speech recognition.

 For the same vowel, differet speakers can distiguished by the features in their glottal waves and formants. 

Even without speech signal analysis intuitively speakers recognize the vowels of their language. That is how those who've never taken any course in speech analysis tell the vowels that are distinctive. 

@ Ruth

You are right. But that's not the question. Jodi asked: How do they do that?

Formant values allow vowel identification, but listeners most probably use the relative (not absolute) values of the first two formants. For instance, the neutral schwa vowel has a F2 equal to 3*F1. The absolute values for adult males are approx. 500 and 1500Hz, depending on the size of the vocal tract. For other speakers (female speakers, children...) the absolute values change, but their relative position of the frequency axis (F2=3*F1) is the important index. This "analysis strategy" could be seen as one of the elements in the process of speaker normalization.

Formant bandwith is proportional to the degree of nasal resonance, so this can also help to identify vowels in languages where nasal resonance is a distinctive vowel feature. Furthermore, loudness ("sonority") can be an index in that open vowels (such as [a:]) typically have more sound energy than closed vowels (such as [i']). 

Format frequencies can't be the entire story, as human perception is relatively insensitive to steady-state stimuli. One can show that this is the case with speech recognition as well (e.g., the inverse-E experiments that lead to RASTA-PLP), although this might apply mostly to consonant recognition.

Thanks very much for these very useful responses.  I'm going to dig into these suggestions and see if I can make some progress on my project.

formants are not the only acoustic features that aid the perception of vowels. pl read through this article. it gives a very good insight into the other factors too. 

Vocal tract normalization by Keith Johnson , Ohio State University  

If you want to do some analysis, the Peterson Barney data set is readily available.

Thanks very much Yaakov.  This was not on my radar.

Vowels are not independent objects, but alphabet elements. To each language and a dialect their relative positioning in formant space is identical both to men and to women and to children but different in absolute frequency space. Thus the person distinguishes at the same time a position of group of vowels in frequency space and sounds.