Hello,
I am trying to go about the training of a feed forward neural network (FFNN) for multivariate nonlinear regression. However, I am not getting satisfactory results in my test set. I am aware of the tricks of the trade, like regularization, cross validation for hyperparameter optimization, etc. I have also seen that some researchers say that FFNNs are good at interpolation when presented with new samples but not so much at extrapolation if the samples in the test set are too different from those in the training set. Here the catch, is that the input-output space of my function is too large to collect a data set that can capture most of it and use it to train the network.
From your experience, what are some actions I could attempt to achieve a good performance in my training set? Any architectures I should try? Also recurrent neural networks are rather tricky to train and keep stable so I rather use FFNNs.
I will appreciate your comments. Thanks in advance for your advice.
Thanks for your answer. By "deep learning" one usually means a lot of inputs (typically hundreds of them), a lot of layers (typically tens of them) and a hell of a lot of data (gigabytes of them). In addition, deep learning was designed initially for classification, not for regression. So just forget about deep learning, design a feedforward net with a single hidden layer (Multilayer Perceptron with one hidden layer). Note that there are scores of published papers on the use of neural nets for inverse dynamics of robotic arms.
In my opinion, the main methodological flaw in your approach is the use of seven output variables, which probably makes the task extremely cumbersome. Designing seven neural nets with 21 inputs and a single output will ease the task enormously.
If you have to build the measurement database, you should resort to experimental planning (also known as "active learning" in the field of neural networks) in order to have a statistically meaningful coverage of the operating region of the robot with the smallest possible amount of data. If the database is already in existence and you cannot make new measurements, you will have to do the best you can, with the available data.
Once the database is available, you have to check the relevance of the inputs, i.e. perform input selection. Are you sure that the positions, velocities and accelerations of all joints are relevant to predict the torque of each joint ? If it is not the case, you may need to find which variables are relevant for the prediction of the torque of each joint ; this is known as "variable selection ». There are a few good variable selection methods, see for instance "Feature extraction, foundations and applications", edited by I. Guyon, S. Gunn, M. Nikravesh and L. Zadeh (Springer-Verlag, 2005), chapter 2 (Assessment methods). Incidentally, the random probe method described in that chapter will find automatically pairs of linearly dependent ("collinear") variables, if they exist, and discard automatically one variable of each pair.
Then you will have to perform model training (use an efficient second-order optimization method such as Levenberg-Marquardt, do not use backpropagation with simple gradient descent), and to perform model selection by hold-out, cross-validation, leave-one-out or virtual leave-one-out. For a full, step-by-step design methodology, see "Neural Networks, methodology and applications", chapter 2 (Springer, 2005).
Remember that feedforward networks can perform only one-step-ahead predictions, i.e. predict the torque at time t+1 given the inputs at time t. If you need predictions on a longer time horizon, you must use recurrent (also termed "feedback") neural nets. For a detailed presentation of the design, training and use of recurrent networks, see book above, chapter 2, section on "Dynamic black-box modeling".
I have same experience. MPL is not found better than neural network with one hidden layer.
Firstly in the task analysis stage, you should be sure the used independent variables are the appropriate ones. If it is lacking independent variables then the quality of the fitting cannot be good, later one you must to look for the appropriate model. You cannot discard the possibility that some regression models structures, including non-lineal ones, by example posynomial and signomial models, adjust better to your data that the artificial neuroal netwowrks. Working on feed forward neural networks you must find the more adequate structure for processes identification. You must try with differente FFNN structures.
Best wishes
Why do you mention "deep learning"? Are you really trying to use deep learning for regression, or are you just trying to train a standard feedforward neural net? What do you mean by "input-output space"? How many variables and how many examples do you have? Just basic questions....
Hello Gérard, thank you for your answer. Well, quite frankly the term "deep" is a bit confusing two me. I have not found a formal definition for it. However, and please correct me I am wrong, it seems to include feedforward neural networks of more than three hidden layers. To answer your second question, yes, I am trying to use the network for a nonlinear regression task, that is, as a function approximator. I want to use this network to model the inverse dynamics of a robot manipulator, where the inputs are the position, velocity and acceleration of its joints and the outputs are the torques at the corresponding joints. There are however infinitely many configurations that can give very different torques, that is what I mean by input-output space and why I said is too large. The input variables are three per robot joint, in my case a total of 21 input variables. Similarly the output variables are 7 (one per joint). I will appreciate if you can give me and advice on how to correctly implement such a network.
You have first to check if your inputs are not redundant, since the lack of performance in multivatiable regression can be due to colinearity between input variables.
Thanks for your answer. By "deep learning" one usually means a lot of inputs (typically hundreds of them), a lot of layers (typically tens of them) and a hell of a lot of data (gigabytes of them). In addition, deep learning was designed initially for classification, not for regression. So just forget about deep learning, design a feedforward net with a single hidden layer (Multilayer Perceptron with one hidden layer). Note that there are scores of published papers on the use of neural nets for inverse dynamics of robotic arms.
In my opinion, the main methodological flaw in your approach is the use of seven output variables, which probably makes the task extremely cumbersome. Designing seven neural nets with 21 inputs and a single output will ease the task enormously.
If you have to build the measurement database, you should resort to experimental planning (also known as "active learning" in the field of neural networks) in order to have a statistically meaningful coverage of the operating region of the robot with the smallest possible amount of data. If the database is already in existence and you cannot make new measurements, you will have to do the best you can, with the available data.
Once the database is available, you have to check the relevance of the inputs, i.e. perform input selection. Are you sure that the positions, velocities and accelerations of all joints are relevant to predict the torque of each joint ? If it is not the case, you may need to find which variables are relevant for the prediction of the torque of each joint ; this is known as "variable selection ». There are a few good variable selection methods, see for instance "Feature extraction, foundations and applications", edited by I. Guyon, S. Gunn, M. Nikravesh and L. Zadeh (Springer-Verlag, 2005), chapter 2 (Assessment methods). Incidentally, the random probe method described in that chapter will find automatically pairs of linearly dependent ("collinear") variables, if they exist, and discard automatically one variable of each pair.
Then you will have to perform model training (use an efficient second-order optimization method such as Levenberg-Marquardt, do not use backpropagation with simple gradient descent), and to perform model selection by hold-out, cross-validation, leave-one-out or virtual leave-one-out. For a full, step-by-step design methodology, see "Neural Networks, methodology and applications", chapter 2 (Springer, 2005).
Remember that feedforward networks can perform only one-step-ahead predictions, i.e. predict the torque at time t+1 given the inputs at time t. If you need predictions on a longer time horizon, you must use recurrent (also termed "feedback") neural nets. For a detailed presentation of the design, training and use of recurrent networks, see book above, chapter 2, section on "Dynamic black-box modeling".
Hello Gérard. First of all, sorry for the late reply. I just wanted to say thank you for your answer. It was very helpful. I have not been able to solve the problem yet, but your comments are certainly guiding me to find a solution.
Once again, thanks for your time and help.
Hello Mr. Garald, as you mentioned regression problem can not be addressed using deep neural network. but there are approaches mentioned by MATHWORKS with time series analysis. i trying to implement times series forecasting using deep neural network, can you guide pls.
Fernando Diaz Ledezma
For regression of non-linear functions NN a nice recommendation is to try first the classic simple feedforward network for fitting the function x^2 and then scale it if you need to deep layers and according to the application.
*FFNN example with pytorch:
class FeedforwardNeuralNetModel(nn.Module):
def __init__(self, input_dim, hidden_dim, output_dim):
super(FeedforwardNeuralNetModel, self).__init__()
# Linear function
self.fc1 = nn.Linear(input_dim, hidden_dim)
# Non-linear
self.sigmoid = nn.Sigmoid()
# Output Linear function
self.fc2 = nn.Linear(hidden_dim, output_dim)
def forward(self, x):
# Linear function # LINEAR
out = self.fc1(x)
# Non-linearity # NON-LINEAR
out = self.sigmoid(out)
# Linear function (readout) # LINEAR
out = self.fc2(out)
return out
# generate the input data
x_vals = np.random.rand(500)
x_train = np.asarray(x_vals,dtype=np.float32).reshape(-1,1)
y_correct = np.asarray([i**2 for i in x_vals], dtype=np.float32).reshape(-1,1)
# instantiate network
input_dim = 1
output_dim = 1
hidden_dim = 20
modelFF = FeedforwardNeuralNetModel(input_dim,hidden_dim,output_dim)
criterionFF = nn.MSELoss()# Mean Squared Loss
l_rate = 0.01
optimiser = torch.optim.Adam(modelFF.parameters(), lr = l_rate)
epochs = 5000
#Training
for epoch in range(epochs):
epoch +=1
#increase the number of epochs by 1 every time
inputs = Variable(torch.from_numpy(x_train))
labels = Variable(torch.from_numpy(y_correct))
#clear grads
optimiser.zero_grad()
#forward to get predicted values
outputs = modelFF.forward(inputs)
loss = criterionFF(outputs, labels)
loss.backward()# back props
optimiser.step()# update the parameters
if (epoch+1) % 500 == 0: # Logging
print('Epoch [%d/%d], Loss: %.4f'
%(epoch+1, epochs, loss.item()))
print('Final - epoch {}, loss {}'.format(epoch,loss.item()))
# Test training
predicted = modelFF.forward(Variable(torch.from_numpy(x_train))).data.numpy()
plt.plot(x_train, y_correct, 'go', label = 'from data', alpha = 0.2)
plt.plot(x_train, predicted, 'b*', label = 'prediction', alpha = 0.2)
plt.legend()
plt.show()
- Badges
- Science topic