Why is the conjugate gradient method not suitable with dropout? Is the conjugate gradient method suitable for denoising autoencoder?

Satyanarayana Vusirikala @Satyanarayana_Vusirikala

07 July 2014 1 7K Report

I was implementing dropout method with conjugate gradient method of updating weights. As I was not getting good results, Hinton sir said that "conjugate gradient is no good with a stochastic method of getting gradients". Can anyone please give explanation for it? Denoising autoencoder is also a stochastic method. Is conjugate gradient method suitable for denoising autoencoder?

Fabrice Clerot

"One of the reasons why conjugate gradient breaks down in a stochastic setting is that the noise makes it impossible to maintain the conjugacy of search directions over multiple iterations."

see

http://research.microsoft.com/pubs/65652/schgra02a.pdf

Badges
Science topic

More Satyanarayana Vusirikala's questions See All

Can anyone please suggest good resources for learning information retrieval

I have been working in artificial neural networks since 6 months. Now, I want to learn about information retrieval. As I don't have any knowledge in it, can anyone please point out some good...

07 August 2014 4,037 1 View

How to apply the k-cross validation technique when the data is in the form time series?

I am dealing with Climate data mining using neural networks. Using rainfall data of 1900 - 1970 year, I have to predict rainfall for 1971 year. In this case, how to apply k-cross validation...

07 August 2014 6,277 11 View

What is the difference between data mining, pattern recognition, machine learning and artificial intelligence?

I am working in artificial neural networks from 5 months. But, up to now I couldn't figure out any difference between data mining, machine learning and artificial intelligence. Can anyone please...

07 August 2014 7,263 5 View

What is the difference between dropout method and adding noise in case of autoencoder?

Motivated by "Improving Neural Networks with Dropout" thesis, In page 2 of the paper, author says that "Dropout can also be interpreted as a way of regularizing a neural network by adding noise to...

07 August 2014 388 3 View

How to perform a stepwise feature selection method in neural networks?

I heard of stepwise feature selection methods for regression problems. How can we perform stepwise feature selection methods when dealing with neural networks? Isn't it computationally expensive?...

05 June 2014 7,303 9 View

How can I measure variance of a model?

Is there a good way to measure variance of model experimentally? I mean, can we give a single real number stating variance of the model? Truly speaking, I want to measure the level of overfitting...

05 June 2014 4,039 4 View

What can I do if a validation error continuously increases?

I am coding a neural network. I am following k fold cross validation technique to avoid overfitting. So, I divided dataset into 10 parts and ran 10 fold cross validation. But, I found that cross...

05 June 2014 9,824 7 View

How to solve overfitting problems when the training set is small?

I am currently coding a neural network on weather data. So, I have only 570 training examples. But, the relation between output and input is highly non linear. How can I reduce overfitting on this...

05 June 2014 1,118 6 View

How to compare two groups with only two measurements?

Hiiiii everyone! I have an enquiry on statistical analysis. I was looking for many forum and it's still cannot solve my problem. I want to compare means of two groups of data but only with two...

03 March 2021 8,796 3 View

What is the most Operating System (OS) which can be INSTALL on Tiny hard devices like Raspberry Pi (RPi) hardware ?

I'm targeting to deploy a mesh network and manually configure MANET routing protocols. I'm preparing scenarios, architectures, and hard devices needed to do that. Are there some step-by-step...

03 March 2021 1,931 5 View

What are some of the research gaps in machine learning and artificial intelligence in africa?

i would to know some of the research gaps in the artificial intelligence field in most african countries.

03 March 2021 6,145 3 View

Enhanced Yellow Fluorescent Protein (Aequorea victoria) DNA sequence?

I am on the lookout for the Enhanced Yellow Fluorescent Protein (Aequorea victoria) DNA sequence. Does anyone know where I can find it? Thank you in advance

03 March 2021 3,568 1 View

What kind of fluid could I use for a pitfall trap that also does not invalidate molecular testing?

Hi, I want to start testing pitfall trap to obtain ants samples, but I need to conduct molecular analysis on those insects. So, what kind of fluid can I use? Ethanol expires too early and I need...

03 March 2021 5,978 5 View

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10?

What's the best way to measure growth rates in House sparrow chicks from day 2 to day 10? Since, the growth curve from day 2 to 10 won't be like the "Logistic curve" it might not follow logistic...

03 March 2021 1,401 3 View

Do we need to write protocol for a second study of a systematic review that has been registered and published?

I have conducted and published a systematic review and meta-analysis research with the topic related to public health and health pomotion (protocol was registed in PROSPERO). Now we want to...

03 March 2021 8,920 3 View

How to increase a model accuracy ?

dear community, my model is based feature extraction from non stationary signals using discrete Wavelet Transform and then using statistical features then machine learning classifiers in order to...

03 March 2021 6,994 5 View

Do I need to do linear regression before running mediation analysis using PROCESS MACRO by Hayes ?

I just wanted to check if I need to run a linear regression separately if I am using PROCESS MACRO to run mediation analysis. Thank you.

02 March 2021 4,359 3 View

Sample dilution for quantitative analysis using ELISA?

If the detection range is in ng/ml but the reference range is in ug/ml for a molecule or protein in serum or plasma .how to dilute and what is the initial volume to be taken for quantitative analysis

02 March 2021 7,670 3 View