I've been working developing and testing blood samples by ELISA since 3-4 years. My experience is that if you try to work with serum could be better to avoid any background due to the interaction with red blood cells.
If you wish to have good and accurate results then ELISA must be performed using serum samples. Whole blood will give you inaccurate results and as rightly mentioned by Sara it may interfere with your final results.