These are the reasons that caused the reduction in ultimate tensile strength of composite materials such as:

(1) Poor dispersion, distribution of the nanoclay

(2) Defect by large particle size of nanoclay

(3) Compatibility of the components in the composite

(4) The matrix system that you used

However, they are not all. Hope this will help

Regards 

I would say a bad interaction between clay (which is slightly polar) with PP (usually non-polar) to be the first explantation to draw.

did you try SEM on fractured samples to investigate the PP-clay interface?

Thank you for your suggestions. This clay was organically modified for non-polar polymers and I also used a compatibilizer (maleic anhydride grafted PP) for better interaction, but still the interaction is bad. I did not try SEM, but i will do those analysis next. It was surprising to see that the Young's modulus was higher than that of pure PP but the yield stress was lower. 

and what about the strain at break?

We are not considering the strain at break / elongation at break for this research work. But those values are very low compared with that of pure PP.

Did you also check by SEM or TEM for how well the clay is dispersed? Perhaps you have agglomerates of primary particles. These agglomerates could be what are actually failing rather than a failure at the clay/polymer interface. 

Also, we found that in some nanocomposites (of other type of nanofiller) with polyurethane, it was noted that for nanoparticles that were covalently modified to be able to chemically interact during the resin cure, the polymer appeared to be UNCURED near the particle surface, resulting in very poor mechanical properties. This strange behavior is under investigation.

I haven't performed SEM / TEM yet. Thanks for the suggestion and will do soon. Also it is interesting to know that polyurethane composites also have such strange behavior.

In such a case, then a bad micro or nano dispersion is surely the core problem. Also worth to mention, 5% hectorite would be a bad idea because of different reasons, have you tried with 1%?

the reason I chose 5% clay loading was due to the capacity of dosage unit in the twin screw compounder. It is difficult for dosing lower clay concentration. Can I say that micro composites are formed rather than nano composites. But still I should need SEM/TEM images to justify this.

These are the reasons that caused the reduction in ultimate tensile strength of composite materials such as:

(1) Poor dispersion, distribution of the nanoclay

(2) Defect by large particle size of nanoclay

(3) Compatibility of the components in the composite

(4) The matrix system that you used

However, they are not all. Hope this will help

Regards