It is very common when the nanofillers don't have very good interaction with polymer matrix.
Without good interaction, the weak van der walls force is enough to bond the fillers to polymer.This weak bonds can transfer the stress between fillers and polymer, therefore show a higher modulus at small strains.
However, when go to high strains, you will find the materials are more brittle, with a low elongation at break. And you will see a low strength.
As Lukas said, without good interaction, the homogeneity of material is low and easy to have crack to break.
There are two possible answers to this question is: 1) because the small particles reduce the mobility of the matrix, which might even develop an interphase around the filler, of a stiffer polymer. This has been widely experienced with very small particle and is one of the main drawback of these fillers (this induces a low impact resistance). You might also 2) end up with a similar problem with bigger particles (higher modulus, lower strain at break) because of the stress reinforcement around the filler. Usually the best tradeoff is with about 1µm particles. This is true with almost all polymers including the "tough" PET and the "weak" rubbers.
You should actually look at the product of the specific surface by the thickness of the interphase. Smaller particles with larger interphase will end up with brittle materials (not tough). You can make an estimation of the fraction of the volume of the polymer that is altered by the presence of the filler. This is a typical size effect (although not wanted !)
It is very interesting to know parameters of your experience? Concentration of particles, polymer type? It is important to know repeatability of the phenomenon, his statistics and a method of definition of the module and strengh