You need a big database and the ground truth (who is the speaker). Then test the threshold level of your technique and measure False Positives, False Negatives and construct a Receiver Operating Characteristic curve. Now, that would be a good starting point :-)
It depends on the application. For example, if it is for a defence establishment, then false negatives may be tolerated, but false positives are an absolute NO, NO! So, the threshold is suitably offset to minimize false positives even at the risk of marginally increasing the false negatives. Whereas, if it is going to enable, say money payment in an ATM, there is less risk (in any case, bank has sufficient insurance to cover the risk) and also, customers will be terribly unhappy if they are genuine, but are rejected by the system. So, false negatives must be minimum, at the cost a few false positives. Then, you adjust the threshold on the other side to get the above desired effect. So, there is no fixed threshold; it depends on what the system designer or the end application demands, in terms of false positives and false negatives.
If you ask the question for a research project, say your thesis, then what mentioned by Fernando Soares Schlindwein is the proper answer in my opinion. Of course, these days the Detection Error Tradeoff (DET) curve is preferred to Receiver Operating Characteristic (ROC) curve. You can find its MATLAB file on the net. On this curve normally the Equal Error Rate (EER) and Minimum DCF are considered as the important measures for comparison of different speaker verification algorithms.
As A.G. Ramakrishnan said "...there is no fixed threshold; it depends on what the system designer or the end application demands, in terms of false positives and false negatives." That is what the ROC curve allows you to choose. :-)
It seems that some colleagues overlooked my previous comment and forget about the importance of the Minimum DCF which is application dependent and the proaility coefficients can be set correspondingly based on the target application, such as defence.