There could be a number of contributing factors for this, including: substrate material, substrate cleanliness, substrate temperature, vacuum conditions, deposition rate, film thickness and initial growth/nucleation.
You can try depositing at a low rate (maybe 0.2-0.5A/sec), I am assuming that you have a good vacuum in the chamber so the mean free path is higher for aluminium atoms. You can also try to heat the substate if possible.