I do not believe that it will work. I would suggest to use either cell sorting or magnetic beads and mark your fibros with antibodies against "fibroblast specific antigen 1" (FSP1) or use Miltenyi anti-fibroblast beads. depending on the cell type you want to work with you should be very stringent to remove the other cell type to get pure cells.
As suggested you could pull-out the fibroblasts, using bead separation. Other markers such as prolyl-4-hydrolase are distinct from smooth muscle actin which can be expressed in both. you might also consider flow cytometry to sort the populations, depending on how pure you need the population. You will probably always have some element of contamination though i.e. not 100% pure smooth muscle cells.