What kind is the data (number of features, number of examples, number of targets)?

set of 100 timestep data to train the network...with set of 5 variables as input and 3 variables as output...give me suggestions....of the 11 days that i am validating the results, some of the predictions are deviating from their targets to the tune of almost 20%...help me out...

Network architecture is 5-12-12-3 and i used 2-2 set of delays for input and feedback regressors 

Thanks

I would not recommend using GA for optimizing a neural net, it will take too long and won't be able to fine tune it.

How did you choose the 5-12-12-3 architecture? You have more parameters than examples, so you should use regularization (e.g. weight decay) or a simpler architecture.

Since you have few examples, don't worry about stochastic gradient descent, you should use a second-order optimization method. I highly recommend Levemberg-Marquadt and Conjugate Gradient.

If you have Matlab, the Neural Network toolbox will offer all those options to you.

Pedro, 

i am already using LM algorithm and i tested the neural network for arriving at the mentioned architecture. How do you conclude that i have more parameters than examples? i have 111 datasets...from which i am using 100 datasets to train the NN and 11 to validate it. Since i need to duplicate the networks, the toolbox does not provide adequate control over the structure of the network.

How to implement weight decay or bayesian regularization? Please share your work if you have prior experience.

Thanks and regards,

Atiya

Atiya, if you have a 5-12-12-3 fully connected network, this means you have 5*12 + 12*12 +12*3 + Biases connections, so you have at least 240 parameters! With only 100 training examples, this means you have a very overcomplete system (in theory).

Weight decay is very easy, you just add a L2 (Euclidean norm) penalty on weight values on the objective function, with appropriate scaling (this metaparameter can be found with cross-validation). If you're using Matlab (or other NN library), that's probably an option. If you're doing by hand, just re-derive the gradient with this additional term.

Another easy way to regularize is to use dropout. It's a recent technique that has been very successful with deep neural networks. You just hide a percentage of hidden units (usually around 50%) every training example, and then you multiply the outgoing weights by after training. This way the neurons are forced to learn independently.

I don't any easy way to implement Bayesian regularization.

I agree with Pedro Tabacof 

I disagree. The minimum number of examples is twice the number of hidden neurons, no matter what is the number of features. There is a paper by Muller that proves it, and my own experience, with many data sets, is confirming it.

I believe that the proposed ANN architecture is too complex - my experience with my training algorithm is that one hidden layer with 5 hidden neurons is sufficient for most practical purposes.

I am ready to  prove it if Atyia can send me his data ( to [email protected]).

Zvi,

I may send you the data but i need solid proof of what you are saying because i need to develop a model for it....and quickly...so please assure me that it would work...i am ready to share the techniques i am using...

Thanks and regards,

Atiya

 Here is a list of my recent papers on ANN:

Zvi Boger Publication List

Updated July 2014

144. P. Elia, S. Raizelman, E. Katorza, Y. Matana, Z. Boger, Y. Zeiri

and P. Mariotini, Bio-markers in urine for detection of cervical cancer.

Presented at the Annual Meeting of the Israel Society for Medical and

Biological Engineering, Haifa, Israel, Feb. 2014.

143. M. E. Lόpez, Z. Boger, E. R. Rene, M. C. Veiga and C. Kennes,

Neural modeling of a one-stage biotrickling filter handling H2S and VOC

mixtures. Journal of Hazardous Materials 269 (2014) 45–55

142. K. Hulkower, A. Sen, Z. Boger, H. Ding and J. Nedrud, Detection

of Helicobacter pylori infection in exhaled breathusing an artificial neural

network in conjunction with a colorimetric sensor array. Submitted for

publication, 2013.

141. T. Kuflik, Z. Boger and M. Zancanaro, Analysis and prediction of

museum visitors’ behavioral patterns types. In A. Kruger, T. Kuflik (eds.),

Ubiquitous Display Environments, Cognitive Technologies, DOI 10.1007/

978-3-642-27663-7_10, Springer-Verlag Berlin Heidelberg 2012. pp. 161-

176.

140. Z. Boger, “Virtual sensor” in industrial control systems by artificial

neural networks modeling. Proceedings of the 2nd

Process Control Conference, Be’er Sheva, Israel, 2011.

139. M. Tasin, A. Lucchi, C. Ioriatti, M. Mraihi, A. De Cristofaro,

Z. Boger and G. Anfora, Oviposition response of the moth lobesia

botrana to sensory cues from a host plant. Chemical Senses 2011;

doi:10.1093/chemse/bjr027

138. Z. Boger, Identification of abnormal behavior in industrial systems

by artificial neural networks modeling. Proceedings of the 1st

Advanced Process Control Conference, Be’er Sheva, Israel, 2010 (in

Hebrew, abstract in English).

137. T. Kuflik, Z. Boger and M. Zancanaro, Analysis and prediction of

museum visitors’ behavioral patterns types. In Minerva Summer School on

Ubiquitous Display Environments, 2010.

136. Z. Boger, Artificial neural networks modeling to reduce industrial

air pollution. In J. Mehnen, M. Köppen, A. Saad, and A. Tiwari, (eds.),

Advances in Soft Computing, Springer, 2009.

Israeli Advanced

Israeli

135. R. Arav-Boger, Y.S. Boger, C.B. Foster and Z. Boger, The use

of artificial neural networks to enhance prediction of congenital CMV

outcome from sequence data. Bioinformatics and Biology Insights vol. 2,

2008, pp. 281-289.

(also available in http://www.researchgate.net/publication_26879608)

134. D. Stopel, Z. Boger, R. Moskovitch, Y. Shahar, Y. Elovici, Using

artificial neural networks to detect unknown computer worms. Neural

Computing and Applications, vol. 18, (7), 2009, pp. 663-674.

133. R. Moskovitch, D. Stopel, Z. Boger, Y. Shahar, and Y. Elovici,

Method and system for detecting malicious behavioral patterns in a

computer, using machine learning. US Patent Application number

US20080184371. 07/31/2008

132. D. Stopel, Y. Elovici, Y. Shahar, Z. Boger and R. Moskovitch,

Improved method and system for detecting malicious behavioral patterns

in a computer, using machine leaning. EU Patent Application number

EP1879124. 01/16/2008

131. J.J. Huseynov, S.B. Baliga, A. Widmer and Z. Boger, An adaptive

method for industrial hydrocarbon flame detection. Neural Networks, vol.

21(2-3), 2008, pp. 398-405.

130. D.C. Meier, J.K. Evju, K.D. Benkstein, B. Raman, Z. Boger, D.L.

Lahr and S. Semancik, Enabling MEMS chemical microsensor arrays

for trace analyte detection. Proceedings of the TRANSDUCERS and

EUROSENSORS '07 - 4th International Conference on Solid-State

Sensors, Actuators and Microsystems, 2007, p 2397-2400.

129. R. Moskovitch, D. Stopel, Z. Boger, Y. Shahar and Y. Elovici,

Method and system for detecting malicious behavioral patterns in a

computer, using machine learning. US Patent Application #20070294768,

2007, EU Patent Application nunber 07001952.6, 2007.

128. Y. Elovici, D. Stopel, R. Moskovitch, Y. Shahar, and Z. Boger,

Improved method and system for detecting malicious behavioral patterns

in a computer, using machine learning. EU Patent Application number

EP1814055. 08/01/2007.

127. Z. Boger: Experience in the applications of artificial neural

networks in bio-informatics. In Meller and Novak, (eds.), Applications of

statistical and machine learning methods in bioinformatics, Advances in

Computational and System Biology Series, Vol. 1. Peter Lang Publishing

Group, Bern, 2007.

126. Y. HaCohen-Kerner, Z. Boger, H. Beck, E. Yehuday, Classifying

documents' authors to their ethnic group using stems and ANNs.

Proceedings of the CAINE-2007: ISCA 20th

Computer Applications in Industry and Engineering, San Francisco, 2007,

pp. 5-11.

125. J.J. Huseynov, S. Baliga, A. Widmer and Z. Boger, Infrared flame

detection system using multiple neural networks. Proceedings of the

International Joint Conference on Neural Networks, IJCNN’07, Orlando,

Florida, 2007

(available also in http//citerseerx.ist.psu.edu/10.1.1.135.6775.pdf)

International Conference on

Zvi Boger Publication List

Updated July 2014

144. P. Elia, S. Raizelman, E. Katorza, Y. Matana, Z. Boger, Y. Zeiri

and P. Mariotini, Bio-markers in urine for detection of cervical cancer.

Presented at the Annual Meeting of the Israel Society for Medical and

Biological Engineering, Haifa, Israel, Feb. 2014.

143. M. E. Lόpez, Z. Boger, E. R. Rene, M. C. Veiga and C. Kennes,

Neural modeling of a one-stage biotrickling filter handling H2S and VOC

mixtures. Journal of Hazardous Materials 269 (2014) 45–55

142. K. Hulkower, A. Sen, Z. Boger, H. Ding and J. Nedrud, Detection

of Helicobacter pylori infection in exhaled breathusing an artificial neural

network in conjunction with a colorimetric sensor array. Submitted for

publication, 2013.

141. T. Kuflik, Z. Boger and M. Zancanaro, Analysis and prediction of

museum visitors’ behavioral patterns types. In A. Kruger, T. Kuflik (eds.),

Ubiquitous Display Environments, Cognitive Technologies, DOI 10.1007/

978-3-642-27663-7_10, Springer-Verlag Berlin Heidelberg 2012. pp. 161-

176.

140. Z. Boger, “Virtual sensor” in industrial control systems by artificial

neural networks modeling. Proceedings of the 2nd

Process Control Conference, Be’er Sheva, Israel, 2011.

139. M. Tasin, A. Lucchi, C. Ioriatti, M. Mraihi, A. De Cristofaro,

Z. Boger and G. Anfora, Oviposition response of the moth lobesia

botrana to sensory cues from a host plant. Chemical Senses 2011;

doi:10.1093/chemse/bjr027

138. Z. Boger, Identification of abnormal behavior in industrial systems

by artificial neural networks modeling. Proceedings of the 1st

Advanced Process Control Conference, Be’er Sheva, Israel, 2010 (in

Hebrew, abstract in English).

137. T. Kuflik, Z. Boger and M. Zancanaro, Analysis and prediction of

museum visitors’ behavioral patterns types. In Minerva Summer School on

Ubiquitous Display Environments, 2010.

136. Z. Boger, Artificial neural networks modeling to reduce industrial

air pollution. In J. Mehnen, M. Köppen, A. Saad, and A. Tiwari, (eds.),

Advances in Soft Computing, Springer, 2009.

Israeli Advanced

Israeli

135. R. Arav-Boger, Y.S. Boger, C.B. Foster and Z. Boger, The use

of artificial neural networks to enhance prediction of congenital CMV

outcome from sequence data. Bioinformatics and Biology Insights vol. 2,

2008, pp. 281-289.

(also available in http://www.researchgate.net/publication_26879608)

134. D. Stopel, Z. Boger, R. Moskovitch, Y. Shahar, Y. Elovici, Using

artificial neural networks to detect unknown computer worms. Neural

Computing and Applications, vol. 18, (7), 2009, pp. 663-674.

133. R. Moskovitch, D. Stopel, Z. Boger, Y. Shahar, and Y. Elovici,

Method and system for detecting malicious behavioral patterns in a

computer, using machine learning. US Patent Application number

US20080184371. 07/31/2008

132. D. Stopel, Y. Elovici, Y. Shahar, Z. Boger and R. Moskovitch,

Improved method and system for detecting malicious behavioral patterns

in a computer, using machine leaning. EU Patent Application number

EP1879124. 01/16/2008

131. J.J. Huseynov, S.B. Baliga, A. Widmer and Z. Boger, An adaptive

method for industrial hydrocarbon flame detection. Neural Networks, vol.

21(2-3), 2008, pp. 398-405.

130. D.C. Meier, J.K. Evju, K.D. Benkstein, B. Raman, Z. Boger, D.L.

Lahr and S. Semancik, Enabling MEMS chemical microsensor arrays

for trace analyte detection. Proceedings of the TRANSDUCERS and

EUROSENSORS '07 - 4th International Conference on Solid-State

Sensors, Actuators and Microsystems, 2007, p 2397-2400.

129. R. Moskovitch, D. Stopel, Z. Boger, Y. Shahar and Y. Elovici,

Method and system for detecting malicious behavioral patterns in a

computer, using machine learning. US Patent Application #20070294768,

2007, EU Patent Application nunber 07001952.6, 2007.

128. Y. Elovici, D. Stopel, R. Moskovitch, Y. Shahar, and Z. Boger,

Improved method and system for detecting malicious behavioral patterns

in a computer, using machine learning. EU Patent Application number

EP1814055. 08/01/2007.

127. Z. Boger: Experience in the applications of artificial neural

networks in bio-informatics. In Meller and Novak, (eds.), Applications of

statistical and machine learning methods in bioinformatics, Advances in

Computational and System Biology Series, Vol. 1. Peter Lang Publishing

Group, Bern, 2007.

126. Y. HaCohen-Kerner, Z. Boger, H. Beck, E. Yehuday, Classifying

documents' authors to their ethnic group using stems and ANNs.

Proceedings of the CAINE-2007: ISCA 20th

Computer Applications in Industry and Engineering, San Francisco, 2007,

pp. 5-11.

125. J.J. Huseynov, S. Baliga, A. Widmer and Z. Boger, Infrared flame

detection system using multiple neural networks. Proceedings of the

International Joint Conference on Neural Networks, IJCNN’07, Orlando,

Florida, 2007

(available also in http//citerseerx.ist.psu.edu/10.1.1.135.6775.pdf)

International Conference on