Supposedly, it's all about background signals if I haven't missed something. Gut feeling (experience from developing ELISAs and LF) tells me that if you add unspecific (but polyclonal) IgG there is some chance of getting a nasty background. I'd rather use BSA or similar (as Xinjie suggested)) and possibly include an non-ionic detergent like Tween 20 for additional blocking.
I agree with Wolfgang, be careful for polyclonal responses to high amounts of blocking with BSA. We prefer BSA over skim milk powder, since it is purified. Try optimising the block incubation and concentration of BSA. Good luck!