This field is not one in which I have worked much as such to help you find a data set is not possible for me. What I can tell you is my opinion and a possible reason for the problem that you are talking about.

Now you would know that in face recognition we save the face of the person in the form of matrices that correspond to the colors found in a certain area and as you know these values change depending on the way the light shines on the face of the person. The problem with black skin is that the values calculated for one person can be similar to that of another person depending on the lighting given that the face structure and a few other factors are similar enough. Because you see in this case we are talking about shades of black and these can be found easily enough on another person with black skin. Of course there are various other factors that matter when talking about face recognition but this is the most basic reason in my opinion for the low performance issue.

Hi! Check out the Face Image Meta-Database (fIMDb): https://cliffordworkman.com/resources/

The fIMDb includes info or estimates on: number of photo sets per source (and numbers of neutral and other sets — e.g., facial emotions), number of subjects per source (with approximate sex distributions), total number of images, approximate number of viewpoints, whether the sources includes photos from more than one ethnicity, whether it includes more than one age group, whether meta-data are available, the photo category (e.g., posed, wild), the reference(s) for the source (e.g. DOIs). I hope this will aid others interested in conducting research on responses to faces.

Thank you Shakti Taneja and Clifford Workman

i found Gender Shade paper, that published by Joy Buolamwini

she believe that races and skin color inequity of train set and test is cause of this bias.

unfortunately could not find the link for download data set

http://gendershades.org/overview.html