You can simply calculate the cartesian coordinates using the crystal matrix A multiplied by the relative coordinates of ALL positions within the unit cell. Then you simply need to determine all distances using a loop. That's what programs like Mercury do. There is no mystery. Of course you need to calculate a bit more than the unit cell content but you can start with the unit cell and then you simply need to check whether there is an atom in a layer of the shortest distance which is even closer to an atom than the shortest distance you've already found.