Can anyone explain difference in algorithm of artificial neural network (ANN) and response surface methodology (RSM)? What are the main differences in their algorithm complexity? How we can compare their algorithm complexity especially when we are using MLP in neural network?
Hi Jamil,
as far as I understand, RSM has a fixed set of basis functions like the several powers of x in an estimated polynomial fit. To adapt such an approximation surface to your data, all you have to do is to select the right linear mixing coefficients of those nonlinear basis functions (there is a one step solution for this problem with solving the normal equation). This might works well with low dimensional data or if you exactly know how to design the basis functions.
ANN on the other side not only trains the linear coefficients in the output layer (in regression problems, this part is equivalent to the RSM), but usually adapts the nonlinear basis functions. Also, you have one type of basis function (e.g. sigmoid or ReLu), unlike the different monomials (different powers of x) in a polynomial. In addition there can be several layers of basis functions which increases the flexibility of the model.
Basically this means, that ANN is an universal approximation without a strict assumption of the shape of the approximation surface.
From the algorithmic point of view:
RSM is probably trained by solving the normal equation (one step solution).
ANN can be trained in many ways. Usually backpropagation, also often in combination with some pre training. The algorithms usually use iterative methods.
Complexity is in both methods dependent on the number of basis functions. In addition you can constrain the complexity by using regularization. Complexity problems can be tackled with hyper parameter optimization (hyper parameters which define the complexity e.g. using a regularization parameter or the number of basis functions). Should be feasible for RSM and ANN.
Hope that helps.
Cheers,
Stephan
- Badges
- Science topic