I had similar problem sometime ago. In the end, I used non-fat milk as blocker overnight. Then wash thoroughly before incubated with antibody/detection agent. Ths Separating blocking and antibody(s) incubation solved my problem.
Be aware that non-fat dried milk contains biotin. If you are dealing with streptavidin-coated plates (?? pure guess!), the sites will be blocked. But why do you need to avoid albumin?