It depends if your protein will bind to albumin. You can remove the IgG from plasma with Protein A/G beads (something like Melon Gel works well too). The albumin you can remove with Cibacron Blue Agarose (http://www.sigmaaldrich.com/catalog/product/sigma/c1535?lang=en®ion=). Unfortunately if you protein binds to albumin the only way is to titrate the plasma and empirically search for a happy dilution. Also Cibacron Blue Agarose will bind other proteins such as nucleotid-dependent enzymes.
Have you filtered your BSA/milk before applying to the membrane? There are Whatman®GD/X syringe filters that you can use to minimize the background interference by albumin.