Compatibility b/w two substances is more often regarded as surface phenomenon. In case of surface functionalized nanomaterials, prediction of compatibility becomes easier otherwise it needs to have long experience of handling composites.

What kind of polymer and nanoparticle are you working with? There is not a general rule about compatibility, instead of it, I would analyze the functional groups that would interact each other.

In most cases, to make inorganic nanoparticle compatible with organic polymer, a kind of surface modiification of nanoparticle is needed. You may compare the polarities of both components, if they are similar you can try to make nanocomposite without such modification.

Please check the attached paper. Also, you would consider the charges on the surface and how this could change with pH.