I built several feed foward neural models on Matlab. It was done with 1 hidden layer and number of neurons on input and output layer was fixed. Neurons on hidden layers was changed from 5 up to 10. The same data set ordered in two differents ways was used. Same run was leaved out for validate on both situations. Results of MSE and R for each neural network are shown in the attached file. The trouble is cannot find correspondence between trends from training and validation.
Hi Roberto,
From what I see, everything's normal.
You mentionned you used validation, so it makes the dots means ?
There is no trend between train and testing...
It is true that a high precision in training will give a high precision in testing.
It is probabilities... You run validation, meaning that there 10 subset created... so there can be a probability that there is difficult subsets (in the created validation set).. and when this happens, it makes the validatio precision lower.
Your curves are what you get with your data.
Machine Learning is just running different techniques with different parameters and select the best performing one.
Sometimes selecting "weird" parameters can give a great precision, just need to try them all.
Yohann
Hi Yohann,
Thanks for your answer. But this validation was made to see prediction capacity of the networks.
Thus the data used didn't participate in training, testing or validation of networks and is actual data from the process.
You are right about that when applied dividerand (like I did) results from training are not necessarily in correspondence with testing and validation steps, because data doesn't have to be the same. But, if you are checking prediction capacity with the same data don't you have to see similar results?
Anyway I will took your advice and take R overall value of training, testing and validation. Maybe trends dependency will be more clear.
Roberto Eloy Hernández Regalado
Can you breakdown how to used the data the generate the training / validation?
training -> 0.90% of data then leave 10% for validation?
"But, if you are checking prediction capacity with the same data don't you have to see similar results? " -> same data? usually the data used to train train, prediction capacity is on the training curve.
validation data is the data unused in the dataset.. 100% - 90% : 10%..
My other question is, to understand,
have you used the full data to train and the same full data to validate the results?
Without splitting 90-10 ?
Yohann
Maybe I didn't make too clear that part.
Breaking down the data:
8 data sets (total)
7 ( training, testing and validation)
1 ( prediction capacity)
To 7 data sets was applied divederand 70% training, 15% testing and 15% validation. Trainlm was used as training function. The results are reported on the graphics
Then networks were simulated using the one data set left and results are also reported on the graphics.
Okay, I understand now. You used validation test set to tune hyperparameters, ie number of neurons in that case.
To me, there should not be a correspondance or a trend correlation between validation and training. So your curves are normal.
are the distribution of your data from training set and validation set comparable?
Yohann
I think the key is last you said. As division for training, validation and testing was made random by divederand.
Then I should try with the overall R and MSE of training, testing and validation. I think this is more representative of the process. Will be more decisive for find out if there is any relationship between the curves.
Thanks for your help
- Badges
- Science topic