Like LDA is linear, ANN is non linear but Decision tree algorithms like CART, Random Forest, C5.0, C4.5,should be kept under nonlinear or a separate category should be made like Decision Tree Classification Techniques.
Attending to the linearity of the solution, algorithms like decision trees are non linear.
They operate in a very different way than other algorihtms like neural nets to minimize their cost function but there are other algorithms that are very different too.
But you can subdivide non linear algorithms in different families (kernel methods, algorithms based on trees, neural nets, ...)
Decision trees work by partitioning the input space in hyper-boxes. They can be seen as a combination of rules, where only one rule is selected for a given input vector.
Finding the optimal tree structure (i.e. the optimal combination of rules) is NP-hard. For this reason, a greedy approach is used to build the decision trees.
That is why, even if decision trees are non-linear, when they are learned with a greedy algorithm, they can not solve the XOR problem.
An efficient way to overcome this drawback is to use random forest. Indeed, when a set of randomized trees vote, they combine variables in a non-greedy way to choose the class.
- Badges
- Science topic
- Similar topics
- Data Mining
- Classification