Suppose that we have a random vector X (with m random variable) and Y(with n random variable). and Y=W.X (single layer neural network). I'm going to minimize the KL divergence between X and Y by gradient descend method. When m=n it's easy and it can be calculated by Jacobean matrix but if m # n, I don't know how to solve it.
Translated into neural network formalism, there are m input units (the components of the input vector X) and n output units (the components of the output vector Y). What's missing is the topology of the network: how the input units are connected to the output units and the normalization of the couplings: do they take discrete, or continuous values? The fact that n is equal or not to m isn't relevant; what's relevant is, whether, in the space of couplings, there exists a solution: a set of, potentially, n*m couplings that can realize the mapping, when X and Y are chosen from a distribution. So the problem, as given, is incomplete.
If the mapping is linear, then neural networks are not relevant. If m isn't equal to n, then the problem is overdetermined or underdetermined and more details are needed to resolve the constraints and/or fix the symmetries.
Another possibility, of course, is straightforward linear regression: Write a functional
S=sum_{m,n}(Y-W.X)^2 and write the least square equations for the (average) values of the W, taking into account, evenntually, appropriate normalization of the variance.
Some computational intelligence approaches are helpful for the gradient descent such as Neural networks, Simulated Annealing, and Genetic Algorithms. Some search-based methods work well as well
- Badges
- Science topic
- Similar topics
- Mathematics