Do you know early stopping? Did you have divided your data in the learning and testing part? You probably have to overfit.

Greetings Bojan

Thank you for your response

I am not sure of early stopping but i sure divided the data into training validation and testing.

This is how i have arranged the data in nmatlab

Input [3x4680]

3 rows are electrode 12,23 and 31

first 1560 columns (52 cycles *30 measurements) = no pressure data

second 1560 columns = pressure on A data

Third 1560 columns = pressure on B data

target [3x4680]

first 1560 columns = no pressure data [100]

second 1560 columns = pressure on A data [010]

Third 1560 columns = pressure on B data [001]

this is the matlab code i implemented

close all;

clear all;

clc;

rng(4151941);

load('NEWCALS1.mat');

input= NEWCALS1(1:3,:);

target=NEWCALS1(4:6,:);

save input.mat;

save target.mat;

x = input;

t = target;

trainFcn = 'trainscg'; % Scaled conjugate gradient backpropagation.

% Create a Pattern Recognition Network

hiddenLayerSize =4;

net = patternnet(hiddenLayerSize);

% Setup Division of Data for Training, Validation, Testing

net.divideParam.trainRatio =70/100;

net.divideParam.valRatio = 10/100;

net.divideParam.testRatio = 20/100;

% Train the Network

[net,tr] = train(net,x,t);

% Test the Network

y = net(x);

e = gsubtract(t,y);

performance = perform(net,t,y)

tind = vec2ind(t);

yind = vec2ind(y);

percentErrors = sum(tind ~= yind)/numel(tind);

% View the Network

%view(net)

% Plots

% Uncomment these lines to enable various plots.

%figure, plotperform(tr)

%figure, plottrainstate(tr)

%figure, ploterrhist(e)

figure, plotconfusion(t,y)

%figure, plotroc(t,y)

Greetings Hilal

Thank you for your time

How do you think I should proceed from here.

What changes in my code would you you recommend.

Since I am new to neural networks any form of help to guide me in the right direction would be appreciated

Greetings Hilal

1. I didn't quite get you when you said start with less number of cells in hidden layer. You mean to say I must reduce the hidden layer size?

2. I have already tried 50%, 25% and 25 % for training, validation and testing and I still end up with 100% accuracy for most values of hidden layer size except from 1-4.

3. How do I normalize the data.

Could you point me to an example that I can refer?

Awaiting a favorable response and thank you for your patience

Hi,

If you visualize your data (in a 3D scatter plot), you'll see they are indeed easily separable. Use something like this :

color = (1*t(1,:) + 2*t(2,:) + 3*t(3,:));

figure;scatter3(x(1,:),x(2,:),x(3,:),[],color)

It is thus normal, and not bad, if you have a very good classification rate, even with appropriate cross-validation. It is just a very easy task for you NN. I also have 100% on a manually extracted test set. I took 3000 random sample for training and the rest for manual testing:

%% extract training/validation set manually

randids = randperm(length(x));

trainids = randids(1:3000);

valids = randids(3001:length(x));

xtrain = x(:,trainids);

xval = x(:,valids);

ttrain = t(:,trainids);

tval = t(:,valids);

%% training

[net,tr] = train(net,xtrain,ttrain);

%% manual validation

yval = net(xval);

figure, plotconfusion(tval,yval)

I did not change your NN parameters. If you have a good classification rate with the validation set, it means it is efficient, simply (if your dataset is appropriately designed of course).

Normalization won't change much here (I tried on your data).

Cheers,

Mickaël Tits

Greetings Mickaël

Thanks a lot for your response , especially with the code . It indeed made a lot of concepts clear for a newbie like me.

Now that i know the method i used was correct, i want to further expand the code for further finding the best H (hiddenlayersize) in case i have some other dataset.

After reading many Q and A fom matlab i have come to the understanding that when

[ I N ]  = size(inputs)      

[ O N ]  = size(targets)    

1. No. of training equations Ntrneq = Ntrn*O

Assuming the default 0.7/0.15/0.15 trn/val/tst

Ntrn = N-2*round(0.15*N)

2. No. of unknown weights Nw = (I+1)*H+(H+1)*O = (I+O+1)*H + O  

3. Mean square error when the solution is approximated by     the target mean

       MSE00 = mean(var(target',1)) % Rsquare = 0

4. Sufficient training goal for the normalized MSE

      NMSE = 0.99

5. OVERFITTING: The number of unknown weights exceeds     the number of training equations.

       Nw > Ntrneq

 This tends to cause poor performance on nontraining data via the phenomenon of OVERTRAINING with an iterative training algorithm.

6. Overtraining an overfit system of equations is mitigated via

      a. Not overfitting Ntrneq >= Nw       b. VALIDATION STOPPING (MATLAB default) 7. No of unknowns is not greater than the number of equations     when

          H Nw ==> H Nw

Hub =( (Ntrneq-O) / (I+O+1))

Ntrials = 10

rng(0)

trueclasses = vec2ind(t);

j=0

for H =round([Hub/10, Hub/2, Hub])

j = j+1

H = H

net = patternnet(H);

for i = 1:Ntrials

net = configure(net,x,t);

[ net ,tr ] = train(net,x,t);

outputs = net(x);

errors = gsubtract(t,outputs);

assignedclasses = vec2ind(outputs);

classerr = assignedclasses~=trueclasses;

Nerr(i,j) = sum(classerr);

% FrErr = Fraction of Errors (Nerr/N)

[FrErr(i,j),CM,IND,ROC] = confusion(t,outputs);

FN(i,j) = mean(ROC(:,1)); % Fraction of False Negatives

TN(i,j) = mean(ROC(:,2)) ; % Fraction of True Negatives

TP(i,j) = mean(ROC(:,3)); % Fraction of True Positives

NMSE = mse(t-outputs)/MSE00

PctErr=100*Nerr/N

end

end

% H = round(Hub/10: Hub/2: Hub)';

% PctErr = PctErr;

% results = [ H PctErr ]

Is it necessary to run a preprocessing on the data,if so what method. Could you please point me in the right direction with the code.

Awaiting a favorable response and thanks a lot for your time.

Some possible causes: a) small dataset; b) a great number of epochs may be causing overfitting; c) features may be linearly separable or problem very easy no NN.

Hi,

In the file you provide, there are only 156 measures (not 1560). The testset is thus composed of 0.15*156 = 23 samples. That is not much for cross-validation.

Anyway, some remarks:

First, the performance you are evaluating (PctErr) should be based solely on the testset. For instance, you can do this: (I did some modifs in your code)

for H =round([Hub/10, Hub/2, Hub])

j = j+1;

%H = H ;

net = patternnet(H);

net.divideParam.trainRatio = 50/100;

net.divideParam.valRatio = 25/100;

net.divideParam.testRatio = 25/100;

for i = 1:Ntrials

net = configure(net,x,t);

[ net ,tr ] = train(net,x,t);

testids = tr.testInd;

testtarget = t(:,testids);

testoutputs = net(x(:,testids));

errors = gsubtract(testtarget,testoutputs);

assignedclasses = vec2ind(testoutputs);

testtrueclasses = trueclasses(testids);

classerr = assignedclasses~=testtrueclasses;

Nerr(i,j) = sum(classerr);

% FrErr = Fraction of Errors (Nerr/N)

[FrErr(i,j),CM,IND,ROC] = confusion(testtarget,testoutputs);

FN(i,j) = mean(ROC(:,1)); % Fraction of False Negatives

TN(i,j) = mean(ROC(:,2)) ; % Fraction of True Negatives

TP(i,j) = mean(ROC(:,3)); % Fraction of True Positives

NMSE = mse(testtarget-testoutputs)/MSE00

Ntest = length(testids);

PctErr=100*Nerr/Ntest

end

end

However, in this code, you are again performing a validation to choose the best H. So, to evaluate the final performance of your network, you should test it on yet other data (as your testset here is used as a validation set for the choice of H).

Another (and quite usual) way to avoid overfitting is regularization. The idea is to add a cost proportional to weights i nthe cost function.

In your case, the problem to model is fairly simple, and involves only 3 features, and you have a sufficiently large number of samples (4680 in your excel sheet). So regularization is not really needed here.

As Eanes Torres Pereira suggests, a much simpler method could work on your problem. See the attached file for a classification based on discriminant analysis (lienar and quadratic), logistic regression, and svm.

Nonetheless the neural net outperforms the rest (with 10 or more hidden neurons), so it is the best method here (you can run the attached script several times with seval random training/validation testsets to convince yoruself).

In your case, all methods seem to work well with or without input data normalization.

Best regards,

Mickael Tits

Greetings Mickaël

Thank you once again for a detailed well written answer. I had made few modifications to the old code.

There are 2 scripts. One is to find the number of hidden neurons (using different training functions [trainbr, train, trainscg, trainrp, trainlm] ) and second is to compare it with other methods [Neural Network, LDA, QDA, LR and SVM] .

I had made a new database with x=[3x156] (x52input.mat contains this new database)

Thus the older database uploaded earlier [3x4680] is null and void.

PDOTPSS_find_number_of_neurons.m 

the script iterates from 5 neurons to 30 at the interval of 5, 5 times each. The resulting figure shows the mean error (of the 5) for trainbr, train, trainscg, trainrp and trainlm.

The number of neurons was chosen to be 10 in this case. (PDOT 5-30 5INT.jpg)

PDOTPSS_compare_classificators.m 

The script runs 20 times and plots the mean of the errors for Neural Network, LDA, QDA, LR and SVM

Errors form figure Pdot compare.jpg proves that NN is the better method.

I hope the algorithm used here is correct.

Zaf

Hi,

First I don't understand very well why you limit your database to 156 sample (which is really not a lot), as I suppose you could easily get more measurements.

With you code for finding the number of neurons, and with your (although null and void) larger dataset, I see that the error clearly descends with the number of neurons. It is not really the case with your smaller dataset.

Note that in your code, you should use a validation set rather than a test set, to make sure that your training function performs a correct cross-validation (to avoid overfitting).

For you comparison script, you forgot to use the training set for the training of your neural network, which will of course give you better (but wrong) results. As a matter of fact, with your smaller DB (156 samples), the NN does not do better than logistic regression (0.5-1.5% error); while with the larger (although null and void) DB of 4680 samples the NN has zero-error.

You can find the corrections in the attached code.

hi?

you should use a validation set rather than a test set, to make sure that your training function performs a correct cross-validation (to avoid overfitting). For you comparison script, you forgot to use the training set for the training of your neural network, which will of course give you better (but wrong) results.