Dear Wim Kaijser

I think It is important to use good, cutting-edge algorithms for deep learning instead of generic ones since training takes so much computing power. Stochastic gradient descent with momentum, RMSProp, and Adam Optimizer are algorithms that are created specifically for deep learning optimization.

5 ways that you can do to improve your machine learning models!

1. Handling Missing Values

One of the biggest mistakes I see is how people handle missing values, and it’s not necessarily their fault. A lot of material on the web says that you typically handle missing values through mean imputation, replacing null values with the mean of the given feature, and this usually isn’t the best method.

For example, imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a missing fitness score. If we took the average fitness score from an age range of 15 to 80, then the eighty-year-old will appear to have a much higher fitness score that they actually should.

Therefore, the first question you want to ask yourself is why the data is missing to begin with.

Next, consider other methods in handling missing data, aside from mean/median imputation:

• Feature Prediction Modeling: Referring back to my example regarding age and fitness scores, we can model the relationship between age and fitness scores and then use the model to find the expected fitness score for a given age. This can be done via several techniques, including regression, ANOVA, and more.
• K Nearest Neighbor Imputation: Using KNN imputation, the missing data is filled with a value from another similar sample, and for those who don’t know, the similarity in KNN is determined using a distance function (i.e., Euclidean distance).
• Deleting the row: Lastly, you can delete the row. This is not usually recommended, but it is acceptable when you have an immense amount of data to start with.

2. Feature Engineering

The second way you can significantly improve your machine learning model is through feature engineering. Feature engineering is the process of transforming raw data into features that better represent the underlying problem that one is trying to solve. There’s no specific way to go about this step, which is what makes data science as much of an art as it as a science. That being said, here are some things that you can consider:

• Converting a DateTime variable to extract just the day of the week, the month of the year, etc…
• Creating bins or buckets for a variable. (e.g., for a height variable, can have 100–149 cm, 150–199 cm, 200–249 cm, etc.)
• Combining multiple features and/or values to create a new one. For example, one of the most accurate models for the titanic challenge engineered a new variable called “Is_women_or_child” which was True if the person was a woman or a child and false otherwise.

3. Feature Selection

The third area where you can vastly improve the accuracy of your model is feature selection, which is choosing the most relevant/valuable features of your dataset. Too many features can cause your algorithm to overfit, and too little features can cause your algorithm to underfit.

There are two main methods that I like to use that you can use to help you with selecting your features:

• Feature importance: some algorithms, like random forests or XGBoost, allow you to determine which features were the most “important” in predicting the target variable’s value. By quickly creating one of these models and conducting feature importance, you’ll get an understanding of which variables are more useful than others.
• Dimensionality reduction: One of the most common dimensionality reduction techniques, Principal Component Analysis (PCA) takes a large number of features and uses linear algebra to reduce them to fewer features.

4. Ensemble Learning Algorithms

One of the easiest ways to improve your machine learning model is to simply choose a better machine learning algorithm. If you don’t already know what ensemble learning algorithms are, now is the time to learn it!

Ensemble learning is a method where multiple learning algorithms are used in conjunction. The purpose of doing so is that it allows you to achieve higher predictive performance than if you were to use an individual algorithm by itself.

Popular ensemble learning algorithms include random forests, XGBoost, gradient boost, and AdaBoost. To explain why ensemble learning algorithms are so powerful, I’ll give an example with random forests:

Random forests involve creating multiple decision trees using bootstrapped datasets of the original data. The model then selects the mode (the majority) of all of the predictions of each decision tree. What’s the point of this? By relying on a “majority wins” model, it reduces the risk of error from an individual tree.

For example, if we created one decision tree, the third one, it would predict 0. But if we relied on the mode of all 4 decision trees, the predicted value would be 1. This is the power of ensemble learning!

5. Adjusting Hyperparameters

Lastly, something that is not often talked about, but is still very important, is adjusting the hyperparameters of your model. This is where it’s essential that you clearly understand the ML model that you’re working with. Otherwise, it can be difficult to understand each hyperparameter.

Take a look at all of the hyperparameters for Random Forests:

class sklearn.ensemble.RandomForestClassifier(n_estimators=100, *, criterion='gini', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=None, random_state=None, verbose=0, warm_start=False, class_weight=None, ccp_alpha=0.0, max_samples=None

For example, it would probably be a good idea to understand what min_impurity_decrease is, so that when you want your machine learning model to be more forgiving, you can adjust this parameter! ;)

Hope this helps.

Regards

You can obtain best fitting model using AIC

Dear Wim Kaijser ,

Methods to Boost the Accuracy of a Model

Add more data. Having more data is always a good idea. ...

Treat missing and Outlier values. ...

Feature Engineering. ...

Feature Selection. ...

Multiple algorithms. ...

Algorithm Tuning. ...

Ensemble methods.

https://www.analyticsvidhya.com/blog/2015/12/improve-machine-learning-results/

Regards,

Shafagat

I'm going to depart from the norm and question the features you're using. What you're counting on implicitly is that your explanatory features are directly/indirectly causally related to the presence of aquatic plants. You seem to suggest that they are causally related to their absence, which isn't the same thing as being related to their presence, and I agree. The conditions you found seem necessary for their presence by the fact that without them they are certainly going to be absent, but you seem to be edging towards the conclusion that this is not sufficient, and I would agree with you there too. What else is required for their presence? Ancestors, for one, to reproduce throughout time. The Absence of competition that will kill them all for another. And probably more I can't think of.

So you seem to be missing causal features linked to the presence of aquatic plants. The "best" algorithms in the universe will inexplicably fail when leveraging non-causal or incomplete causal features. More over, with a complete set of causal features that suffers from "significant" data bias may also lead to inexplicable model failures; this could also be your case.

Unfortunately entropy dictates that we may not always have the information we need to make reliable predictions/models, and in this case it's never the algorithms fault nor can they overcome this limitation. But if your goal is to make a model that reliably predicts the presence of aquatic plants, then your best bet is to discover more causal features to complete your set of causal features.

Dan Harris One of the best answers so far. Unfortunately, suggestions and especially data on more features seem very limited :(. I also tend to think that spatial autocorrelation plays a huge role, (e.g. if a species occurs upstream than it is also likely to occur downstream), seedbanks, ability too reach locations, flood pulses and other hydrological characteristics. These are extremely difficult to define, but the would be viable options.

The question I am struggling with: Is a model that is very good in predicting TN but "horrible" (perhaps it improves with more data???) in predicting TP just as useful considering this is all we have? As such, if an "absence" is predicted and somehow we want/value a species to be present than there is something "wrong" and we have an educated guess why this species might be absent. However, if an presence is predicted, but a species is not there, we would not exactly know why. A bit of the issue is that than still the same variables are blamed (it is a bit like a witch hunt).

In some sense this all seems reasonable, but the laws currently in place focus on at least 2 of these general variables. I am just a bit stuck and often come to opposite conclusion to what others suggest.

You can optimize the parameters using GridSearch in Python, take a look at this scikit learn documentation: https://scikit-learn.org/stable/modules/grid_search.html

Wim Kaijser, definitely pursue the options available to you.

The "better than nothing" evaluation is always a consideration worth making. The value of the of a model depends on the context. I know of many models that are excellent at TN but not TP in classification contexts, and its wonderful--Covid tests are useful in the opposite extreme. The key is to not ask more of the model than the features can provide, and therefore, focus on imaginative ways to use what you've got. What else can you do until you complete your set of causal features (and harvest the data)?

The witch hunt might end if you change your perspective on what your model is doing. What if you viewed your model as a hospitable predictor rather than a presence predictor? So if you desire a certain species, then your model can tell you if conditions are ripe for it to thrive (don't be surprised if they are there already); or if the conditions need to change before it can thrive. Is this valuable? You tell me. Why do you need the model in the first place? What function is it serving and why?

"I am just a bit stuck and often come to opposite conclusion to what others suggest", story of my life, brother. It's part of being a card carrying member of the independent thinkers club.

Dan Harris Not really sure if "need" seems to be an appropriate term for these models. They are a summary of my work and I still hope that with more data the performance becomes better as the features are fixed (https://snwikaij.shinyapps.io/RF_macrophyte_models/). The models also show that general patterns observed by many are indeed there, but seemingly these are not enough to suggest a species should be present if a "critical limit" is not surpassed. I thought that somehow I would be able to boost performance, but also many other ML types (Naive Bayes or K-NN) reach similar or worse results. Besides that, the idea behind the algorithm is beautiful and easy conveys a message as they are quite pragmatic compared to an article or long report.

You can increase your accuracy by changing hyperparameters. Try to change a little the values and then not the results u will understand the things

You can try diffrent activation function of the model;

• ‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x
• ‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).
• ‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).
• ‘relu’, the rectified linear unit function, returns f(x) = max(0, x)

Also, try different slover functions;

• ‘identity’, no-op activation, useful to implement linear bottleneck, returns f(x) = x
• ‘logistic’, the logistic sigmoid function, returns f(x) = 1 / (1 + exp(-x)).
• ‘tanh’, the hyperbolic tan function, returns f(x) = tanh(x).
• ‘relu’, the rectified linear unit function, returns f(x) = max(0, x).
• Note: The default solver ‘adam’ works pretty well on relatively large datasets (with thousands of training samples or more) in terms of both training time and validation score. For small datasets, however, ‘lbfgs’ can converge faster and perform better.

Try using smote to balance the data and then using feature engineering and grid search cv. Go for ML models first then probably deep learning

… Read more

Share

Aashna Ahuja I already applied undersampling, oversampling, SMOTE and ROSE. The ROSE function in caret has some issues with "," and oversampling performs worse than under sampling and SMOTE. I am a bit warry of NN as I find them somehow complex to interpret. Otherwise, for RF models I do not have to transform my variables. I think I am limited by the features which is an result as well, but I need more rigorous, cross-validation and testing.

A basic way to improve ML performance is HPO (hyper-parameters optimization). you can do this using a RL or BO as well. There are also many open-source codes around there . Maybe this https://towardsdatascience.com/10-hyperparameter-optimization-frameworks-8bc87bc8b7e3 will help you.

Dear Wim Kaijser

This paper can help you

Article Advanced Meta-Heuristics, Convolutional Neural Networks, and...

Wim Kaijser I can now reference my paper on the causal approach to ML since it is now published. https://ieeexplore.ieee.org/document/9438325

Depending on the amount of data you have, classifying 48 species of plant might be difficult. Is it possible to bin species at the genus or family level?

In practice grouping indeed seems to work better. You can group them based on known "preferences" to the predictor variables or co-occurrence. However, I find it incredibly difficult to find clear discriminatory differences among most species (some are indeed more easy to discriminate e.g. water lily and mosses). Predictions/patterns between the majority seem to be less clear than often is described, in my opinion. Although experts indeed highlight patterns in quite descriptive ways, and some patterns are quite generalizable, they seem relatively variable. If I try to enumerate/predict in more detail, it becomes quite tricky. At least, I feel uncomfortable/uncertain, to make strong claims/conclusions.

There are two options to look at this, predicting each species individual or discriminate between groups of species. Still, predicting presence/absence is discriminating between two groups. In practice discriminating between 3-5 groups seems reasonable (both descriptive and to predictive) over broad spatial scales and conditions. Yet, performance of more than 3-5 groups falls around(ish) a Cohen's kappa an accuracy ~0.4 and accuracy of ~60%. A simple reason is that clear groups/clusters are absent. In the Fisher iris dataset clear groups can be observe along the different vectors (ecologist would possibly say gradients/variables). But, in reality these clear groups are not really there. I often have some "acceptable" arbitrary cut-off based on Cohen's kappa (0.3). I do not create more groups if kappa drops below 0.3. However, anyone working with other data will laugh his ass off as 0.3 is so incredibly low. I have seen people complain about a kappa of 0.65.

Some have tried to fit expert judgment to the discrimination of 5 groups (actually only 4) with an LDA, see "From expert judgement to supervised classification: A new approach to assess ecological status in lowland streams" by Baattrup-Pedersen (2013). According to this results in a mean Kappa of 0.21 (I calculate 0.16 based on Tab 3.). Nonetheless, the experts seem satisfied. Also, predicting individual species, see "Variable importance for sustaining macrophyte presence via random forests: data imputation and model settings" by van Echelpoel and Goethals (2018) result in a kappa of 0.2-0.3. Another study predicts groups in lakes, but I forgot the actual setup of the methods I am not sure if they first group based on the variables and then also predict the groups with these (circle reasoning), it is still an interesting article though. I think they had 8 groups and kappa of 0.4 and accuracy of 50% (see "Prediction of macrophyte distribution: The role of natural versus anthropogenic physical disturbances" by Bertrin et al., 2018).

However, the marginal performance can indicate multiple things 1.) the data is too noisy (normal or by sample design), 2.) ecologist accept more marginable limits due to complexity (I can also select two groups resulting in a kappa of ~0.6 and accuracy of ~85%), 3.) the wrong predictors are used (mostly the rationale behind predictors is weak). 4.) the wrong model setup are used. But, I am do not know which one ¯\_(ツ)_/¯, kind of frustrating.