I want to train two deep neural networks on two different data sets. The aim is same in both ( predicting cancer relapse) but data sets contain different type of information. I am looking for the best way to combine these two networks to be able to predict the outcome more accurate.
Two straightforward methods are combinations from the beginning and from the end.
To combine from the beginning, you can mix the data sets and train a new network on the new data set. The new network should be able to grasp knowledge embedded independently in different data sets. But the former architectures and parameters are no longer crucial.
To combine from the end, you have to propose some kind of polling mechanism to use the output of both networks given the same input. However, I doubt whether such a polling/selecting mechanism works. Suppose you train 5 networks on independent data sets, after well tuning, they should be able to grasp knowledge from the data fed to them only. But to predict from a new input might concern knowledge/pattern similar to only 1 in 5, therefore it is not determined that the majority always has a privilege. It is some sort of policy-making problem which is more involved.
What you want to achieve might be the combination of different networks from the layer of architecture and parameters. I believe that this is impossible. Not mentioning the difference in architectures, the identification of the "same" network from parameters itself is impossible to be done (consider the numeric issue and symmetry in parameters). So I suggest you use either of the methods proposed above and refer to some decision-making/mixture-of-experts algorithms.
Hope it would be helpful. : )
Thanks Fangqi Li for your suggestion. The first method doesn't work for me since data sets have almost 22,000 feature for each observation. So, combining them together maybe makes data much more complicated. I go for the second method to see if it works or not.
Combining from the beginning is probably the most efficient way. For this, you can use the Expectation-Maximization algorithm (EM algo) and a mixtures of experts approach. See for example:
M. I. Jordan and R. A. Jacobs, "Hierarchical Mixtures of Experts and the EM Algorithm," Neural Computation, vol. 6, pp. 181-214, 1994. DOI 10.1162/neco.1994.6.2.181
http://www.cs.toronto.edu/~hinton/absps/hme.pdf
Yes you can.
There are three ways I can think of, depending on your requirement.
I have only written a few papers on NNs, but why not cross validate on the same data and pick the NN with the best cross validation score?. Use that with the appropriate data. As it is now you don't seem to know that net A is even acceptable on data B. Have I missed something? David Booth
- Badges
- Science topic