Antimony (Sb) can be toxic, if enters human body is large quantities via inhalation or ingestion (for example, some reports indicated that ingesting 0.53 mg of Sb per 1 kg of body weight produced violent vomiting in humans; you can check the data on levels and types of hazardous exposure on line). However, antimony is also used medicinally to treat humans infected with parasites. Interestingly enough, pigs are not affected by antimony levels that are hazardous to humans. Most likely, Sb used for alloy modification is not going to be easily released into environment in hazardous quantities, since it is added to alloys in small amounts, and is tied with other metals in the form of solid solutions.
It is dangerous to eat Sb. In Al alloys you should check its boiling point (reduced upon alloying), and the vapor pressure above the liquid alloy upon its production. Probably it is mostly dangerous at a highest temperature due evaporation. Second, it might be dangerous if it appears in soil and dissolves out. But it is not realistic, as majority of Al alloys are recycled, and also Al has good corrosion resistance, protecting Sb from dissolution.
Sb is safe in the solid solutions and alloys. The antimony may use under manufacture of the p/n junctions in the solar cells which are non-toxic in principle.