Hello,

I think the best to try first is SVM ! Forget MLP , it is "has been" now for the majority of problems like yours (OF COURSE not for very complex problems with huge amount of data, HERE the new revolution is DeepLearning, which use MLP)

This is simpler because you do not have to define a priori the structure of the net as you mentionned (layers, units ...), and the weight initialisation is not sensitive.

It will be necessary to adapt your data, chose a strategy for multiple output problem (one/one or one/other, your can try the two solutions), and beginning by a linear kernel, make a n-fold generalization procedure, and if it is not efficient you will try polynomial kernel.

You will also be obliged to try different value of the C parameter (it is a parameters to avoid overfitting, but it is difficult sometime to find a fine tuning, try first with C in [0.8,1.3], the first try C = 1 simply)

The SVM works largelly better when you have normalized data, but in your case if you have only boolean data there will be no problem.

(just centralyze the data by substracting 0.5 to have data -0.5 0.5, and you can also multiply after by 2 to have your boolean data as -1 / 1, I think it will be more efficient)

If it not the case I suggest to you to centralize/normalize using the min-max method (substract the mean and divide by the abs(max)).

From my experiments it works much better.

Get the software SVMLight from Thorsten Joachims at CORNELL

(very easy to compile)

http://svmlight.joachims.org/

Sincerely yours,

I could give you more information is you can explain with more details :

- your inputs (reals, binaries ...)

- your target (number of classes)

- and the number of sample (compare to the all possibilities, if finite !)

Hervé

================================================

PS = You have also a solution with is mine, which could be also efficient for this problem, it is a constructive algorithm but optimizing the GAP (like SVM, so the generalization ability), here is the url of the mains relevants papers : to be read in order ;-)

Article A New Threshold Unit Learning Algorithm

Article Training a Neuron in Sequential Learning

Conference Paper A Constructive Algorithm for Real Valued Multi-category Clas...

If you cite me please cite the first in the list (and not the first papers I have made for WCNN95 in DC)

BUT I think definitively that SVM should make the job well/better and without a complicate programming job !

Hello,

I think the best to try first is SVM ! Forget MLP , it is "has been" now for the majority of problems like yours (OF COURSE not for very complex problems with huge amount of data, HERE the new revolution is DeepLearning, which use MLP)

This is simpler because you do not have to define a priori the structure of the net as you mentionned (layers, units ...), and the weight initialisation is not sensitive.

It will be necessary to adapt your data, chose a strategy for multiple output problem (one/one or one/other, your can try the two solutions), and beginning by a linear kernel, make a n-fold generalization procedure, and if it is not efficient you will try polynomial kernel.

You will also be obliged to try different value of the C parameter (it is a parameters to avoid overfitting, but it is difficult sometime to find a fine tuning, try first with C in [0.8,1.3], the first try C = 1 simply)

The SVM works largelly better when you have normalized data, but in your case if you have only boolean data there will be no problem.

(just centralyze the data by substracting 0.5 to have data -0.5 0.5, and you can also multiply after by 2 to have your boolean data as -1 / 1, I think it will be more efficient)

If it not the case I suggest to you to centralize/normalize using the min-max method (substract the mean and divide by the abs(max)).

From my experiments it works much better.

Get the software SVMLight from Thorsten Joachims at CORNELL

(very easy to compile)

http://svmlight.joachims.org/

Sincerely yours,

I could give you more information is you can explain with more details :

- your inputs (reals, binaries ...)

- your target (number of classes)

- and the number of sample (compare to the all possibilities, if finite !)

Hervé

================================================

PS = You have also a solution with is mine, which could be also efficient for this problem, it is a constructive algorithm but optimizing the GAP (like SVM, so the generalization ability), here is the url of the mains relevants papers : to be read in order ;-)

Article A New Threshold Unit Learning Algorithm

Article Training a Neuron in Sequential Learning

Conference Paper A Constructive Algorithm for Real Valued Multi-category Clas...

If you cite me please cite the first in the list (and not the first papers I have made for WCNN95 in DC)

BUT I think definitively that SVM should make the job well/better and without a complicate programming job !

Thank you sir,

The answer seems perfect to the problem at hand. I would want to discuss it further.

1. What neural network architecture should we use? How to decide that? and how to justify the use?

Answer: you should begin from the simplest MLP with just one hidden layer and compared to another classifier for the baseline e.g. k-NN, SVM, naïve bayes, etc. The baseline is a justification why you use the particular machine learning

2. Is it possible that one neural network activates two different neurons in the final output because I want it that way?

Answer: Do you mean the NN has multiple outputs? If yes, It is possible to deploy several nodes in the output layer. This architecture usually utilizes for multiclass classification which each neuron represents the probability of each classes.

3. Initially, how do we assign weights to the connections in neural network?

Answer: This article may good for you

https://ac.els-cdn.com/S1474667017419380/1-s2.0-S1474667017419380-main.pdf?_tid=2e518e74-d974-11e7-84d1-00000aab0f26&acdnat=1512448048_5e1a72e6712b14e709822ba757ebb228

4. Is there any tried and tested formula to determine the number of hidden layers and number of neurons in each layer in a neural network?

Answer: There is no definite rule to determine the number of node in the hidden layer. Several studies have been performed to find the optimum number of node. One article which can be considered to solve this issue:

https://www.hindawi.com/journals/mpe/2013/425740/

5. how to chose an activation function and how to justify its use?

Answer: There are several activation functions which possible to use. It depends on the case and the data characteristic. Logistic and hyperbolic tangent are the commonly used activation function. For multiclass classification, you can use Softmax function in the final layer which represents categorical output for more than two classes.

6. Is it possible to combine the output of two different neural networks ?

Answer: Yes of course, you can construct ensemble classifier based on the outputs from several classifiers. Then, the voting mechanism is used to determine the predicted class.

Respect sir,

This is very insightful, I was able to clarify all my doubts with your answer. thank you so much.

2. Is it possible that one neural network activates two different neurons in the final output because I want it that way?

Actually, all outputs are activated, they are giving you probabilities (depending the configuration?) for each class. ie for output 1 [0.95 0.01 0.01 0.01 0.01]. You can have less discriminated results if two classes are close.

So if I understand well your question, I would use probabilities as outputs and choose the two first.

Hello,

First of all, I think that the data base should be analyzed with an AFD,

just to see if it is close to linearly separability problems or not.

(there can have some classes completely separated from the others using just an hyperplan !). AFD is a good tool for that. It is a technic from "factorial analysis". A linear technic which search in the n-space the best rotation to increase the distance between all clusters (all classes samples). You can visualize that viewing "the principals plans".

You can also have a primary overview using xgobi for example. It is very instructive. You can view projections of your data in 3d, and varying the angles of rotation have a look how are close your clusters.

For the multiclass problem the best to my mind is to make n SVMs, 1 class separating the n-1 others, and after that you fuse your results, using ony boolean operations. In case where you have ambiguity you need to compute the soft-max of the output of the n SVM, and it will give you probabilities, as said in past answer. Then you choose the highest.

I think that the software I mention do that (not sure, but I guess, I do not remember).

And secondly, I disagree with Sir Dedy Rahman Wijaya (sorry ;-),

even with the multiclass problemematic, SVM are much more efficient and easy to use, while having very much controlable generalization power : it is an intrinsic property of SVM due to the Vapnic theory of Structural Risk Minimization. It converge to what we call the OMC (Optimal Margin Classfier) which is almost the optimal.

I say almost because in fact a great number of persons think that this is the optimal in the bayesian meaning BUT IT IS NOT. The optimal in the bayesian meaning is the center of gravity of what we call the "version space" (it is a polytope which represent all the solutions), and the best generalization is the center of gravity. (the version space is a compact).

In this version space the OMC is the center of the maximal embedded hypersphere in the polytope. AND the difference could be non negligeable.

It depends on the geometrie of the polytope, which is a consequence of the patterns distribution.

SVM minimize only the RM (Strutural Risk), not the Bayesian Risk, and for ending this comments, Bayesian Risk is the best we can do ! (see short description of Bayesian Decision Theory).

Sincerely yours.

PS = You can also try another algrithm which converge also to the bayesian optimal which is the "Forest Trees", even using weak classifier like simple percpetron. It could give very good results. (In the case you have also to hae the same strategy n times one against the others)

Hervé

Just yesterday, I realized the following:

If you are expecting for your NN to compute a double inference, we achieved much higher result accuracy by splitting the problem into individual inferences and building two different NN for each inference. In our case, we had temporal, environmental conditions that were used to predict labeled product outputs. The first inference was to build a Long Short Term Memory Network (LSTN) that focused on temporal aspect of data and predicted the future environmental conditions. The second was just a simple ANN that was used the predicted future environmental conditions and predicted the product outputs.

This example also reflects the aspect of how much training/validation data you have available. If we had decades of training data, a single deep LSTN would probably perform well. The bottom line is that less data you have, more engineering falls onto your shoulders -- building custom kernels (seems like a long forgotten art, pre-processing data with moving averages, etc etc)

Otherwise the comments and suggestions in the previous threads are right on.

Good luck

Recurrent neural networks must be used to model a dynamical system. The reason is that they contain self-feedback loops in the form of weights that manifests as a memory to the neural network. They will also require less number of inputs to approximate any nonlinear dynamical systems. If the system is static then a static feed forward neural network can be used (like RBFN, MLFFNN). Here also Recurrent neural network can be used

Hi,

The answer to your 2nd question : Is it possible that one neural network activates two different neurons in the final output because I want it that way?

Yes it is very much possible to do so. Give two labels(the one that are the answers) to your data set. All neural networks work on the concept of cost functions and back propagation. So keep the expected o/p as 0 for others and 1 for those two labels. Back propagation (which happens when you train) will now adjust the weights so as to maximize value for those two labels for each column of your training data set.

Answer to your 3rd question : Initially, how do we assign weights to the connections in neural network?

There is no way we assign the proper weights to the connections. The term "learning" comes for this specific part of neural network. All our model has to actually perform is find the right weights for our data set (so that the o/p matches with the expected o/p) and even expect to handle unknown data sets.

Answer to your 5th question : 5. how to chose an activation function and how to justify its use?

a = sum of (xi * wi ) + b

Sigmoid gave some importance to negative 'a' values. Range is [0,1]

Tanh gave equal importance to 'a' and '-a'. i.e. the range is [-1,1]

And if you want to completely discard these negative 'a' ReLu is used where all negative 'a' are 0 and all +ve values are mapped from 0 to 1.

Answer to 6th question : If you want to process these two neural networks side by side then u may need extra computational units like high speed GPUs. If u want to just pass the value then RNN are the best examples where images are identified using CNN and then a caption / phrase is generated using LSTMs.

Hope this helps.