I am looking for coupling agent that give a good dispersion of the filler in polymeric matrix and a good bonding strength between the filler and the matrix
You may try the same silanes which are suggested for glass fibers. They will probably work due to the similarities in surface chemistry. You should take into account, however, waht kind of resin you want to put your treated filler in. In epoys resins epoxy or amine functional silane is suggested, in unsaurated polyester acrylic function should be chosen.
You can chose an appropriate alkoxysilane to graft on your nanofillers with specific terminal functional group hydrophillic/hydrophobic as per requirement . The choice will also depend upon in which polymer you want to disperse your nanofillers as suggested above. Even functional oligomers which have similar properties of the polymer in whish you want to disperse them will also serve as suitable surface modifiers but their choice will again depend upon your polymer.
Perhaps in addition to the good suggestions above you might, as an alternative, consider how montmorillonite clay is modified with organics so that it can be exfoliated and used in a Nylon matrix. For instance, see:
http://www.mdpi.com/1996-1944/3/4/2588/pdf
You will not be getting exfoliation, but the modification of silicate surfaces with organic/hydrophobic groups might give you the surface modification needed for processing into a polymer -- if you chose the organic portion of the modifier with that desire in mind. Various modifiers are proposed these days for montmorillonite, so it is not an instance of just one chemistry being effect at dispersing the clay fillers, I believe.