You can use not necessarily DMSO. If your compound is soluble in water or PBS, or you may it dissolve in a cell culture medium - you may use them. When you test liquid plant extracts, usually your solvent of active substances could be ethanol, methanol, acetone, etc. DMSO is usually preferred for chemical compounds, as it dissolves many substances and also it is not evaporating fast (after you dissolve compound, usually you keep that solution for experiments even more than several months or years; also you perform experiments at 37C temp, and it enhances evaporation of many solvents).
The problem is that DMSO is toxic for cells at higher concentration, we do not use higher than 0.5%, but try to keep it as low as possible, sometimes 0.1-0.25%. Also, if you want to compare different compound activity, you should use the same DMSO concentration between them, as DMSO also helps to penetrate compounds through the cell membrane, and thus it could influence the activity.