different optimizer available sgd, rmsprop, adagrad, adam, based on decide with dropout layers you require with loss functions.

Dear Mahesh Mahendrakar,

Thanks for the answer, but they all find the optimum learning rate. I want to find the optimum number of hidden layers, neurons and learning rate! is there any toolbox or code available for this purpose?

Hi Mohammad

Our group has conducted several researches about how to find the optimum number of layers and neuron. You can see our one is built upon DAE (fixed depth) and another one is built upon DNN (having different depth). Basically, we develop a method to measure the network capacity. You can see it in detail here

1. Article DEVDAN: Deep evolving denoising autoencoder

2. Chapter Autonomous Deep Learning: Continual Learning Approach for Dy...

we also made the code publicly available, the link is written in the paper.

You can use grid search.

First, you have to define parameter space (for ex. number of layers [2-5], number of neurons for each layers [10-100] ).

and then, You will try each parameter set step by step and list accuracy.

This method is greedy. Maybe take some time. But at the end, it will return optimal parameter set.

Tuning hyper-parameters can be automated, and if your approach requires tuning a considerable amount of hyper-parameters it is better to automate the tuning process using a standard technique such as Bayesian Optimization. The Scikit library offers the scikit-optimize (skopt) function that allows you to easily proceed with this.

If "Bayesian" becomes a bit overwhelming, you can also resolve to performing hyper-parameter tuning using easier methods like;

1. Grid Search: https://scikit-learn.org/stable/modules/grid_search.html#exhaustive-grid-search

2. Random Search: https://scikit-learn.org/stable/modules/grid_search.html#randomized-parameter-optimization

3. Gradient-based optimization: https://arxiv.org/pdf/1502.03492.pdf

Another tool that proves to be useful with beginners is Talos (https://github.com/autonomio/talos).

Dear Andri Ashfahani

Thank you so much, I'm studying your papers on DEVDAN & ADL. Pretty amazing works indeed.

Dear Manuela Jeyaraj

Thanks for your help. Unfortunately I'm not familiar with Python, is there any MATLAB toolbox or code made with the same purpose? If not what is the best and fastest way to learn Python?

Hi Mohammad Sadegh Ayubirad ,

If you are working with neural networks, python is the most powerful language that you need to opt for. Scikit is a python library that you can simply call for before beginning your script. In order to install this script, you'll need to first install the library using a pip install sklearn.

However, since you mentioned your unfamiliarity with python itself, I think you can gain some insight or a quick-fix walk-through on machine-learning mastery (https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/).

There are some other great blog posts and articles on the "Towards Data Science" medium publications such as (https://towardsdatascience.com/how-to-build-your-own-neural-network-from-scratch-in-python-68998a08e4f6) and (https://towardsdatascience.com/how-to-build-a-simple-neural-network-from-scratch-with-python-9f011896d2f3).

The above are only guidelines on building your neural network using python. Once you get a bit more comfortable with python, you can consider optimizing your network with the hyper-parameter tuning techniques that I mentioned in my previous reply.