The distance matrix contains 18 Gb? Well then you definitely need more than 4 Gb. A process can't access all of the memory available on the machine due to usage by the OS and other necessary processes, so I would say you should probably find a machine with 32 Gb of memory. You don't quite need that much but usually you don't see anything in between 16 Gb and 32.
About this question, I have a solution that I would like to share with you. For those who use the Mothur software and if you have obtained after the formation of the distance matrix a large file and you could not turn the cluster command to grouping the sequences in OTU . In this case, you can choose the threshold present (cuttof) in the command below according on the capacity of your computer: dist.seqs (fasta = datafinal.fasta, cutoff = 0.10, processors = 4). In my situation I used cutttof = 0.10 instead of cuttof = 0.15 and it worked well