Unfortunately, I'm not sure there is ONE good answer to your question. I would assume that you will find more diversity the farther apart the samples are taken - but that is just a hypothesis - and it has been disproved in my own work before. My studies in subsurface coal and shale seams have revealed that the similarity in microbes between samples across varying distances is wildly different across various geographical locations. In some basins, we could sample over thousands of feet and get the same microbial community whereas other basins had a different microbial community every foot. Without knowing your study goals, and how distance could alter the variables important to your study, it's hard to know which distance makes the most sense.

Here are two examples of strong variation on the mm-scale:

http://www.nature.com/ismej/journal/v6/n6/abs/ismej2011181a.html

and http://www.nature.com/ismej/journal/v7/n1/full/ismej201279a.html

Matching sample size to niche-defining gradients is crucial.

The question has been a hot topic for some years in soil microbial ecology some years back and the answer was quite surprising. In fact, the different studies pointed out a larger heterogeneity at the local scale (due to local aggregates, availability of nutrients, etc.) than at the global scale, for which larger samples are homogenized. So at the cm scale diversity may appear larger than at the km scale. If you look at the work from the Tiedje or Simonet labs, you should find a few references on that.

This has some limits though since at the mm a fraction of mm scale, you have a more homogenous context and the diversity will drop drastically, making comparisons.

In sediments, microbial populations will be influenced by fluids circulation which may impact the local populations drastically. So the same observation might hold as well : a somewhat similar diversity/structure on the large and global scale, and a much more divergent picture at the local scale.

The last is what we observe in shales… with one ore thing to consider : What is conserved at the local or large scale is the "function" of the microbial population, rather than it diversity/microbial populations, e.g. you'll find methanogens or sulfato-reducers wherever the conditions are appropriate, but they may differ from on setting to the other.

I would not expect much change at the Km scale unless you change habitat e.g. hydrothermal vents or a whale carcass nearby. Most of the variations found when comparing samples derive from incomplete coverage. My prediction would be that as more and more sequence is done results of biological replicates will be more similar (and less important to have them).

Very interesting. I work with macrofauna burrows in marine sediments. Microbial communities vary greatly on the mm or cm scales here. The entrenched paradigm argues that there is no effect of geogarphical scale on microbial diversity- everything is everywhere due to unlimited dispersal capacity. I argue the importance of microniches. See also literature by Fenchel and Finlay.

Of course, but micro niches will be the same on average if your sample is large enough. On the other hand, sediment is very heterogeneous. This is one reason I work with the water column. It also has microniche diversity, but a more predictable one.

No because different macroorganisms appear to harbor different microbial communities in their burrows. Interesting don´t you think?! All microniches are not the same.

As mentioned by Cassandra most likely not a single answer to that question. Based on literature and own experience I would say that depends on the degree of depth variation in the geochemical composition of the sediment. A homogenous geochemical profile is likely to be reflected in a homogenous microbial community structure.

So in my opinion the answer to your question depends on the nature of the sediment.

Here is a paper dealing with that issue:

http://www.pnas.org/content/109/42/E2846/1