I'm attempting to design a neural network to approximate SED fitting.
I have a set of 20,000 runs through the SED fitting program, MAGPHYS.
Each run contains:
- A set of 43 input features, which contains 21 values representing a histogram of observed radio fluxes over wavelength and 21 associated values representing signal to noise ratio for each flux value, as well as 1 redshift value
- 32 useful output features (there are other outputs from the system, but I don't need them).
I've built a neural network in Keras to attempt to learn this function. Currently I'm using 40 hidden nodes per layer and 4 hidden layers, in addition to one input and one output layer. The input and hidden layers are all using TanH activation functions and the output layer is using a Linear activation function.
I am normalising my input and output data to within 0 and 1 using minmax normalisation.
I've tried a lot of different combinations of neural network parameters, such as:
- Different optimisers (Stochastic gradient descent, Adagrad, Adam etc.)
- Different learning parameters (momentum, learning rate, weight decay)
- Different number of hidden nodes and different number of hidden layers
- Using a TanH output layer
- Normalising my input data between -1 and 1 instead of 0 and 1.
- Using only the 21 flux readings without the signal to noise ratios (reducing the inputs to the system to 22 rather than 43)
Regardless of any of these parameters, the network always seems to output values that are very close to the averages for each of the 32 outputs. Sometimes the network will output exactly the same number for every test, or sometimes these values will vary slightly, but will still be a value very close to the average.
What would cause my neural network to always output values like this?
Am I doing something wrong with my network design, or is there something else that I'm missing?
Is there anything I can try in an effort to get my network to actually learn this function correctly?
http://www.iap.fr/magphys/magphys/MAGPHYS_files/readme.pdf
My Opinion:
Assume we are working with a data set and we are expecting a ANN to approximate it into a line. When our data is scattered all over a 2D space the ANN might approximate it as a point rather than a straight line. This is kind of over fitting attempt by the network.
It is important to select the features that better defines the expected approximation. Instead of a random reduction of input features try selective elimination.
OR
The target function is almost impossible to approximate.
Try a separate ANN for each output
Try backpropagation (if you're not doing that already) and inspect the weights at each epoch to see how they are changing
Four hidden layers seems very complex, try fewer hidden layers - set out a ystematic experiment trying different numbers of neurons in a single hidden layer, then add further layers based on your findings
If you're not confident with the tool, try building a very simple ANN to validate your approach to the tool (exclusive OR)
We found summarising time series data with skew and kurtosis useful in clasifying oil rig corrosion (unpublished undergraduate project).
If your fluxes are real numbers try banding them to improve separability
Sorry if these are things you've already considered, I can't tell from your post.
You're most likely not training long enough. Your gradients will be very small, you are not escaping from some saddle points in your error surface because your errors all average out. Apart from training longer, you can:
- Use the ReLU activation function to get faster convergence;
- try harder at network initialisation (see, e.g., http://neuralnetworksanddeeplearning.com/chap3.html#weight_initialization)
No idea what SED is, but take care your data set is also large enough. I presume you use mini batches? See also a bunch of other hints at the youtube video I posted below---but it looks like you have a good approach already.
https://www.youtube.com/watch?v=8hAGqGKb-Xk
PS once your network starts to learn, you'll need dropout to prevent overfitting.
I recommend the following way:
1. use Support Vector Machine to try first.(I recommend libsvm)
2.use Restricted Bolzmann Machine to initialize the weight, then stack the NN and do backpropogation.
3.use batch normalization to train NN.(this technique was proposed in 2015)
Once you use Keras, you can watch the demo code of the Keras.
Hope this useful.
Try reducing the number of hidden layers. Your network is probably going into overfitting....similar to rote learning.
Since you said that the "the network always seems to output values that are very close to the averages for each of the 32 outputs". I think you are using too many layers and the network is an inadvertently duplicating the input ("mirroring the input")
My guess is:
(1) Overfiltting.
But since you probably have very many samples this may not be the cause.
(2) Redundant information in input data.
There is too much redundant information in your input vector, all components are not independent. Therefore to extract the number of "independent variables" from the input data you need to reduce its dimension.
You can have a look at the following they may be useful:
http://arxiv.org/ftp/arxiv/papers/0911/0911.0225.pdf
and some of our other papers on dimension reduction available on ReseachGate
-----.
I hope I am helpful.
- Badges
- Science topic