Yes. Chloroform will have activity too. once you have the organic phase you have to evaporate the chloroform. It is hard to perfrom this experiment cause the organic compounds extracted by this method won't diffuse easily in the agar. I would add the organic phase to a liquid medium, then evaporate the chlorophorm by heating, and measure the yeast growth by optic density. of course you make a control with chloroform to check that possible remnants are not affecting the yeast growth. Another option would be to ad the aliquot of chlorophorm to a disk of paper and to evaporate it... but again, chorophorm extractives don't difusse easily.

I agree with Marc that Aspergillus fumigatus is a poor source for the discovery of new activities. But independent from the scource of the extract I would not use chloroform for the bioassay. Use methanol or dimethylsulfoxide. The high boiling point of DMSO is a disadvantage but DMSO is usually well tolerated by eukaryotic cells.

You can used Ethyl acetate for extraction of secondary metabolites and DMSO for checking the bio activity of secondary metabolites against Candida albicans because methanol may show zone of inhibition.