I have cleaned the data for developing countries. Now the problem is there are 13 variables and 77 countries. There is inconsistency in the data. For all variables, the data for few countries are missing for the whole time period or few years out of the selected time frame. How to deal with this. Removal will lead to a dataset of hardly 15 countries with full data.
Depends which additional covariates you have to model your target variables. If I understood it well, there is no additional observed covariates? (and you cannot do a feature engineering to create some?)
If so,
- countries with only missing: exclude them. There is no way to impute them (expect you have additional observed covariates)
- countries with a few missing: impute them, e.g. using R package VIM and function kNN (as a first guess).
Kind regards, Matthias
MISSING DATA means the data is not there. To make such data "appear" by whatever means calls into question if the validity of the work and possible ethical issues with the research. The best way to deal with countries with missing data is to eliminate them. If after elimination you end up with 15 countries with full data---it means a research based on 15 countries with substantiated evidence. You have a better and more solid ground to base your argument on full data.
DATA REPORTER such as the IMF, or World Bank dealt with countries that lack data collection or untrustworthy reporting by leaving the data blank or as missing---they do that for a reason. Since the data you used is already secondary data, the best way to deal with missing data is to avoid manipulation by creating numbers that did not really exist. Leave such countries out until such time the data collecting authorities see fit to report them. They did not report them either because their are not available or available but is not trustworthy.
Paul Louangrath Thank you, sir. What is your view about imputation. If I have data for Bhutan for 5 years out of the 12 years, should I execute it? And then leave out the countries which do not have any data for any of the year?
Matthias Templ, Thank you, Sir. Even after imputation, the panel data is not balanced. A few of the variables have no data at all. If I remove them, the number of countries will go down. I hope I could execute both imputation and then removal of those countries where no data is available.
@Nikee Silayach .... I would advice against data imputation for the following reasons:
1. Mean imputation reduces variance of the inputted data and cause bias.
2. Mean imputation reduces standard errors. As the result this may invalidate hypothesis testing.
3. Imputation may lead to loss of relationships among variables.
Other than technical problems mentioned above, imputed data to cure missing data may raise issues of data integrity and ethical questions on the part of the writer.
In your case (Bhutan) you have 5 years of data out of 12byears period, the missing period is 7 years, i.e. you have less than half of the data need for the study. By imputation, your would have come up with "synthetic data" for 58% of total data set for Bhutan. Delete the country or find other variables that have adequate data for this country.
Paul Louangrath ok Sir, I will simply remove the countries with missing data. Thank you so much.
You may constitute an imputation strategy to replace missing values. For this, you have a wide range of univariate/multivariate statistical and machine learning alternatives. You just have to choose the method you prefer depending on the constraints (e.g., Gaussian distribution function, compositional nature, etc.) and assumptions (e.g., spatial similarity) on your dataset.
For practical examining of these, you may refer to R packages such as "accelmissing", "ForImp", "imputeMissings", "missRanger", and "VIM".
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude