This probably has to do with the fact that variance is sensitive to outliers. What you are looking for are so-called "robust statistics". There is a rather decent Wikipedia article on such measures: https://en.wikipedia.org/wiki/Robust_statistics.

Median absolute deviation and inter-quartile range are some of the robust alternatives to variance.

Presumably to "scale down" by a factor N means (1) to reduce the number of pixels in the image by a factor of N, simply by throwing away pixels. You are not, for instance, (2) smoothing the original image with an N-pixel sliding window and then down sampling the image, or (3) applying a median filter with an N-pixel sliding window and then down sampling the image.

For case (1), which I assume is your case, the change in variance will be most sensitive to the content of the image.

For case (2), you should be able to predict the change in the variance; I guess on the order of a factor of 1/N of the original variance. Prior smoothing may not be desirable.

For case (3), you might also see a predicable change in the variance.

In any case, the point is that rather than invariance of the variance with scaling, you might find a predictable rule for the change in variance with scaling, by applying a suitable operation to the image during the scaling process.