Using HSA/BSA/Albumin in ELISA you are able to avoid non-specific interactions, eg. between serum IgG and plastic surface in ELISA plates. This is crucial step in this type ELISA.

The incubation buffer (for samples and secondary antibodes or conjugates) has three major tasks:

1. keep the conditions despite of the sample dilution constant. Therefore a dilution higher than 1:10 is mostly helpful. You can see the effects in experiments named "dilution linearity". Samples in different dilutions should give you the same results after recalculation with the dilution factor. In a normal assay it's needs to get validated. see also ICH guidelines in the link. In higher dilutions the impact of the protein concentration of the sample gets smaller due to the high and constant concentration of your inert protein. 

2. Avoid unspecific reactions. That's done by the detergent, by increased NaCl-concentrations (0.3 mol/L instead of the normal 0.15 mol/L)

3. keep optimal conditions for the immun reactions (pH, ions, ...)

Just one last remark: HSA can contain IgG with a wide range of specificites except to use a highly purified (and that's expensive!). This can disturb the assay as well. Therefore I prefer to use a hydrolysed gelatine (Gelafusal), which is a commonly used drug in emergancy medicine. It's used as plasma expander and should be available from pharmacies. 

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf

Depending on what the ELISA is intended to detect, HSA could interfere as Stephan notes. If you are using it to detect human antibodies, I would not use HSA as the buffer since it could potentially interfere. If you are doing a sandwich ELISA with a non-human reporter antibody, then the HSA probably won't interfere. In addition to hydrolyzed gelatin, other common reagents are non-fat milk protein and Bovine Serum Albumen (BSA).