Sintering is driven by the surface free energy of the powder particles. This energy may be orientation dependent. Therefor the initial contact between two particles can have different driving forces. However in cubic crystal systems this anisotropy is not very pronounced.
More important is the orientation difference between two particles in contact. If there is no orientation difference the two particles form a single crystal. The contact will grow with time, but no densification is possible. The compact will remain porous, only the morphology of the pores will change.
Since the orientation of particles in a powder is random, normally there is a difference in the crystallographic orientation between the powder particles. A high angle grain boundary will be formed an act as sink for vacancies arriving from the concavely curved neck surface. The annihilation of these vacancies at the grain boundary means sintering with decreasing porosity. At the same time the contacts grow.
If some of the grain boundaries are small angle or special boundaries, they cannot annihilate vacancies. At these points the compact does not sinter. But the surrounding overcompensates this with shrinkage.
The role of crystal orientation was studied allready in the 70ies by Werner Schatt (see e.g. https://doi.org/10.1002/crat.19750100808)
In many cases the powder particles are polycristalline. This means, that in one particle contact many different orientations create sinks for vacacies and promote shrinkage.
For more details literature is recommended. One should go back to 50ies, 60ies and 70ies for this aspect.