I think it is not enough to have 4 values in one set and compare it with other set of 37 values. Is their any practical difficulty in getting more number in first set with 4 values? 

Mann-Whitney is not testing equality of medians.

The Stata Journal (2012) 12, Number 2, pp. 182–190

To my knowledge there is no simple test for that hypothesis. You may consider a bootstrap technique.

EDIT: See below, I found "Mood's median test".

For testing of equality of medians, even chi-square test can  be used provided there are enough numbers (say more than 5 )in each cell. Consider the common median and find out the frequencies below and above for both the categories considered.  

Ramnath, I looked it up and in fact, there is a test for equality of medians:

Mood's test for medians

This is based on a chi-squared test on the numbers of values above and below the global median. If the chi-squared test is substituted by Fisher's exact test, the test will work even in cases with cells having expected counts of less than 5.

Thank you giving this hint!

No, i can not increase samples size. But, thank you very much for your suggestions. After doing Fisher's exact test using values above and below global median, i got the p.value = 0.047 (just significant).

You have 41 observations; how did you decide in which category to put the observation with the median value?  :)

This is Mood's Median Test, with the caveat that usually the test is formulated to count "median or lower than median" in one group, and you may have thrown the median value in with the other group.  But the test formulation for this is arbitrary.

@Salvatore S. Mangiafico: Thank you for your reply. Indeed they are already categorised.

Let me rephrase my comment:  Since you have 41 observations, one observation is equal to the global median.  If you include that observation with one group, you get a p-value < 0.05.  If you include it with the other group, you get a p-value > 0.05.  If you include it in neither group, you get a p-value > 0.05.

This is a quirk of Mood's Median Test.

But this should give you pause with the reliability of your result.

I would look at bootstrapping also.