i cant help you in pop gene but suggest you to use other software for population genetics like ARElQUIN, GenAlex. Use CONVERT Software to convert your data into different formats. Search Google for Convert. it is a nice population genetics software and do most of the analysis which you require

how much memory has your computer? Anyway since it's an old program there should be no problems with memory (at that time 16 MB was common), therefore it might be a problem with running it on a modern computer system (?)

You should ask directly the developers at http://www.ualberta.ca/~fyeh/popgene.html

in the hope that it's still an active website

And yes if nothing helps use a more modern program.

Its happens due to wrong way of data input. No for lack of memory

I am having this problem with GenAlEx, actually. My format is correct and can run many of the tests, but some do not finish and tell me "Out of Memory" even on a computer with 32GB of RAM. Any suggestions? Thanks.

From my experience, I think some of those downloaded software are not properly updated and may not run effectively on modern day system. Out of memory error message was not from computer but due to algorithm error in the running of the program. You may re-download the GenAlex or you try Arelquin. I hope it serves your purpose 

"OUT OF MEMORY" always shows up when the data input file does not meet the requirements, in all the cases it has happened to me - it was a space/gap left at the end of the row

Thank you. I had same problem With popgene , It solved by removing end of the row extra spaces

I faced this problem before and solved it by using ''saving as'' to save the content in PopGene32 at first , and opened that file again.