You are comparing apples with faucets (i.e. no real link at all). XRD is not TEM. They're measuring totally different things. This is so basic that no text book or reference will suffice. BTW, XRD (phase analysis/lattice parameters) is different from XRD line broadening used (Scherrer equation) for crystallite size even though the same equipment can be used for these measurements - you should be careful with your terminology. Perhaps a better question to formulate would be 'Under what circumstances could we get possible agreement between X-ray diffraction line broadening results (XRDLB) and TEM data?'
Crystallite Size - size of a single domain with the same orientation
Therefore, particles typically consist of multiple crystallites.
You are inherently measuring the different definitions above by using different techniques, and therefore MUST differentiate between them. TEM, specifically STEM (s is for scanning), can get you overall particle size. Diffraction within the TEM can give you lattice parameters and in turn the unit cell, however, this is fairly time consuming, complicated, and out of the scope of your question. XRD, by whichever analysis technique (Scherrer, integral breadth, Rietveld, etc.), you are measuring the scattering domain size, i.e. crystallite size.
The condition when particle size = crystallite size is when the particles are single crystals. In my personal experience, this condition rarely occurs. I've measured multiple nanoparticle systems on substrates and compared with XRD, and "agreeable" values are normally 10-30nm different. The reason for this deviation is the inherent broadening of nanoscaled materials in the XRD pattern. Also, in any type of microscopy, there is a limited sample size of particles, therefore counting statistics are poor.