The origin of loss of ductility is primarily due to intergranular fracture. When such fracture occurs, polycrystalline material shows a severe brittleness & poor ductility, irrespective of whether they are metallic, ceramics or intermetallics.
Ductile materials show work hardening (their strength increases with plastic deformation). This effect avoids localization of strain (as the surroundings of a strained area has less strength). When the strength of a material is increased, the strain hardening effect is reduced which results in an earlier onset of strain localization (thermal softening winning from strain hardening). This results in less strain to failure, hence a more brittle material. Also at higher stress levels other failure mechanisms may become active, also resulting in the reduction of failure strain of the material. Example of this is the Ductile-Brittle-transition-temperature (DBTT) below which a ductile material (increasing its strength with lower temperature) suddenly fails as a brittle material.
Brittle materials as we experience them at room temperature (RT) have a DBTT above RT and can become ductile when heated above their DBTT.
"Why polycrystalline material have a tendency to get more brittle when their strength is increased?"
In general, all crystalline solids embrittle when they become harder. If you strengthen a single crystal (e.g., via solid solution strengthening), it also becomes more brittle (the fracture toughness goes down). The same is true for a polycrystalline material if you strengthen it by, say, work hardening, or grain refinement. The simple explanation is that the size of the plastic zone goes down and that leads to a smaller fracture toughness.
Now, what you write afterwards:
"The origin of loss of ductility is primarily due to intergranular fracture. When such fracture occurs, polycrystalline material shows a severe brittleness & poor ductility, irrespective of whether they are metallic, ceramics or intermetallics."
This does not make much sense and is a bit confusing. First, you have to distinguish between ductility and fracture. Fracture can happen by inter or intragranular crack propagation, but ductility is not necessarily associated with a specific fracture mechanism. Ductility tells you how much deformation a material can withstand without failing, how it fails, that's a different issue. Then, when fracture occurs the material fails. So materials cannot show a severe brittleness and poor utility WHEN fracture occurs. It's actually the opposite: fracture occurs IF materials have high brittleness and poor ductility.
Jaime Marian answer fully covers your question. It is noteworthy that in polycrystalline material, by designing the grain boundaries it is possible to simultaneously increase strength and ductility (e.g. nanotwinned material)