The origin of loss of ductility is primarily due to intergranular fracture. When such fracture occurs, polycrystalline material shows a severe brittleness & poor ductility, irrespective of whether they are metallic, ceramics or intermetallics.
Ductile materials show work hardening (their strength increases with plastic deformation). This effect avoids localization of strain (as the surroundings of a strained area has less strength). When the strength of a material is increased, the strain hardening effect is reduced which results in an earlier onset of strain localization (thermal softening winning from strain hardening). This results in less strain to failure, hence a more brittle material. Also at higher stress levels other failure mechanisms may become active, also resulting in the reduction of failure strain of the material. Example of this is the Ductile-Brittle-transition-temperature (DBTT) below which a ductile material (increasing its strength with lower temperature) suddenly fails as a brittle material.
Brittle materials as we experience them at room temperature (RT) have a DBTT above RT and can become ductile when heated above their DBTT.
Jaime Marian answer fully covers your question. It is noteworthy that in polycrystalline material, by designing the grain boundaries it is possible to simultaneously increase strength and ductility (e.g. nanotwinned material)