Since I wanted to compare males' scores and females' scores with in treatment group. Which of the following tests should I use: Wilcoxon Test or Mann Whitney U test? I am in doubt.
I'm making the assumption that you cannot run a parametric test because of violations. Based on your description, if you could run a parametric test would it not be a 2-way ANOVA (treatment group and sex group on scores) rather than a t-test. In this case, there are several options and the selection might depend on what software program your are using. The most standard is Friedman's nonparametric test (see SAS code here: http://support.sas.com/documentation/onlinedoc/stat/ex_code/132/friedman.html ). There is also the Scheirer-Ray-Hare extension of the Kruskal Wallis test, although there has been some criticism of it recently. If you are looking for a nonparametric method equivalent to a t-test, then the Mann Whitney U is best for your data since it appears that you have independent data for the groups.
I agree with James - if you are simply comparing two independent groups (males/females) on data that are non-normal then a Mann-whitney is the way to go. However, it is not clear whether you might also have 1) two or more treatment groups, or 2) whether you are wishing to compare pre/post scores by gender. If you have large enough samples you might consider for 1) a factorial ANOVA or 2) a doubly repeated measures ANOVA OR an /ANCOVA with pre-test scores as a covariate and post test scores as the DV. Of course, in this last, gender would be the IV. If robust, your results should be OK despite the possible non-normality of the data. Of curse, the analysis depends upon your design. Hope this adds to your solution. Kate