I hope this is an answer to your question.

I believe Goodfellow et al ( Goodfellow, I. J.; Le, Q. V.; Saxe, A. M.; Lee, H. & Ng, A. Y. Measuring invariances in deep networks. Advances in Neural Information Processing Systems (NIPS), 2009 ; Goodfellow, I. J.; Courville, A. & Bengio, Y. Joint training deep Boltzmann machines for classification International Conference on Learning Representations, 2013) have shown that "with increasing depth, the representations learned can also enjoy an increased degree of invariance".

My understanding, in plainish English, is that whilst lower-level representations are not strictly rotation-invariant, at higher levels the network 'learns' that certain representations are simply rotated versions of each other.

Any help?

Dear Primoz,

I can't talk about Boltzmann machines in particular, but I hope that my understanding of neural networks and deep learning will be of help.

In a perceptron, neurons from one layer are "connected" to neurons in the next layer, and so on from input to output. The connections between adjacent layers in a perceptron network is an all-to-all, whereas deep learning emphasizes a hierarchical organization.

Now, what does "connected" mean? In a perceptron, a neuron will sum all the outputs of its input neurons and apply an activation function, usually a hyperbolic tangent. Since summation is commutative, you can permute your inputs and still get the same output, which is a kind of invariance.

With deep learning strategies, you establish a hierarchy of classifiers that act first on a very local level. Then each classifier learns a specific feature of the local input and outputs a more abstract summary, with  less detail, for the higher levels to process. So you have successive abstractions when data go from bottom to top which kind of allows nodes of upper levels to reason about concepts, instead of details. This, coupled with commutative operations such as summation, endow your network with a basic ability to learn invariants.

There are other models that provide explicit mechanisms to learn invariants, though, such as HTM (see link attached).

I hope this applies to Boltzmann machines as well.

Cheers,

Fernando

https://en.wikipedia.org/wiki/Hierarchical_temporal_memory

If you rotate your images and train any model on that, obviously it learns about rotation. I don't know any method that can force RBMs to be rotation invariant but there are methods for shift invariant RBMs like Andrew Ng's Convolution RBMs which has probabilistic max pooling which in theory makes it shift invariant. 

Cheers,