.

a good starting point :

http://www.uvm.edu/~dhowell/StatPages/More_Stuff/Missing_Data/Missing.html

also, before embarking into "imputation of the missing values", do not forget to review the relational model(s) your data table comes from ; missing data may be a direct consequence of the flattening process of a relational model : if a client does not subscribe to service S, it is logical to find her invoice value for service S "missing" and there is no reason to impute anything there, not even a "0" !

.

Fabrice, hi,

thanks for your answer!!

I will check the link up and see how I can proceed.

Do you know references or do you have ideas of missing data handling in case there is a mixture of data type (continuous and categorical)?

.

Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models

Nicholas J. Horton and Ken P. Kleinman

The American Statistician, 61(1), 2007

Abstract :

Missing data are a recurring problem that can cause bias or lead to inefficient analyses. Development of statistical methods to address missingness have been actively pursued in recent years, including imputation, likelihood and weighting approaches. Each approach is more complicated when there are many patterns of missing values, or when both categorical and continuous random variables are involved. Implementations of routines to incorporate observations with incomplete variables in regression models are now widely available. We review these routines in the context of a motivating example from a large health services research dataset. While there are still limitations to the current implementations, and additional efforts are required of the analyst, it is feasible to incorporate partially observed values, and these methods should be utilized in practice.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1839993/#!po=42.5000

.

Although not many details regarding your problem are given, I am guessing you are trying to learn a classifier over your data. If that is the case, a classic approach would be to use the generalized Expectation Maximization (EM) algorithm (http://cs229.stanford.edu/notes/cs229-notes8.pdf , http://med.stanford.edu/tanglab/courses/gen245/Lectures/Lecture_03.pdf , etc.). There are loads of tutorials online.

Hope this helps

George, thanks for your answer.

I might have omitted the goal that indeed is the classification. I still do not get how to deal or start applying EM to GMM for the following case: Matrix of n x p (n instances, p features), and in all n instances there are missing data in one or another p. The source of missing data is not randomly but depends on other features values (that I do not know)...how do I start?...thanks again!!

The missing data components are not a problem. Generalized expectation maximization will account for them.

Here is what you need to do to get started:

Assuming a datum is of the form, (x1, x2, ..., U, ... , xp) , where U is an unobserved component, and that you have n such observations and k classes {C1, C2, C3, .., Ck} to which they may belong.

1) Define a joint likelihood distribution, p( x1, x2, ..., xp | Ci) in parametric form. If x1, ... , xp are real variables, then a typical such joint can be a multivariate Gaussian with mean μ and covariance matrix Σ (μ and Σ are the parameters that EM will try to estimate). If, like you mentioned, some components of your data vector are not real, then you may adjust your joint (i.e., consider a multivariate Gaussian for the real variables independently, and some other discrete distribution for the rest and assume independence in order to obtain the joint as the product of the two distributions). Assign values for your covariance matrices and means (initial guesses).

2) Assign prior probabilities p(C1), p(C2), ..., p(Ck) for your classes. These are initial "guesses" for the EM algorithm. In the total absence of knoweledge, assign 1/k to all of them.

In other words, the marginal probability p(x1, x2, ..., xp) is a so-called "mixture model":

p(x1, x2, ..., xp) = p(x1, x2, ..., xp | C1)p(C1) + ... + p(x1, x2, ..., xp)p(Ck)

3) After (1) and (2), the only "unknowns" are the missing data components in your data vector. In order to obtain good "guesses" for these unknowns, you will have to run inference over these variables per data instance. In other words, given your parameters, find which values of the missing variables maximize the marginal distribution p(x1, x2, ..., X, ..., xp). Note here that this step should be repeated inside the EM iteration after the parameters of the model have been re-estimated.

4) After you have filled-in the missing data, you can run the EM for a sufficent number of iterations.

5) When done, to classify a datum (d1, d2, ..., dp) you simply have to find a class index m, such that:

m = argMax {p(d1, d2, ... dp)p(Ci) , over all i = 1, ..., k}

I hope the above helps. There are many details, unfortunately, but this is more-less the way to do it. I believe one of the best texts I have read on generalized EM is by "Probabilistic Graphical Models" by Daphne Koller.

George, hi,

many thanks!!! I will try the algo you described and see what are the results.

Another successful method are self-organizing maps (SOMs, introduced by Kohonen) or topology representing networks (google for Thomas Martinetz) as robust neural maps for data representation. In our group we usually aply these networks with some intelligent interpolation schemes additionaly incorporated ...

I confirm that self-organizing maps (SOMs) are good candidates. Furthermore, it is also the case of neural gas nets and probabilitis neural networks (PNNs)

HI Thomas, Patrice, thanks for your comments. As far I understood, SOM uses distance to detect the activated neuron and then to update the neurons around the activated. Since I have missing data with values -1 or -256 for data in range 0 to 1, would it bias the SOM? moreover, the data contains categorical values. Distance among categorical is not quire representative, isnt it? what are your suggestions to deal with them using SOM?

If you are after classification: Learning Vector Quantization can be done without imputation of the missing values. You can just omit the missing components when calculating distances. At least for data sets with not too many missing values that's a good start and it is very easy to implement.

In view of various advice given towards handling your problem, I think a comprehensive understanding of missing value imputation will greatly help. One such article is "Missing value imputation for gene expression data: computational techniques to recover missing data from available information", by Alan Wee-Chung Liew, et. al. Briefings in Bioinformatics. Vol 12 No 5, Dec 2010. Despite its specific application area, it successfully summarizes almost all known methods of imutation.

.

with SOM (or neural gas etc), when you have missing data, the basic treatment is

- compute distances with the non missing components only

- select the winning neuron from these distances

- update the winning neuron and its neighborhood with the non missing components only

see for instance

Self-Organizing maps for processing of data with missing values and outliers: application to remote sensing images

Bassam Abdel Latif and Gregoire Mercier

Abstract

This chapter presents how to recover a data set that contains missing values, errors and outlier values using the Self-Organizing Maps (SOM). It has been shown by many authors that if a data set contains missing values (missing components of some observations), then the SOM is a good candidate to recover it. The idea is as simple as to use the center of each subclass to estimate the missing values of a given observation. The virtue of the SOM regarding this problem is two folded: firstly, it is a non-parametric regression procedure that does not suppose any underlying models of the data set, and secondly it uses the information from similar observations to refine the positions of subclasses centers and hence gives better estimation.

Therefore, the SOM for missing value will be detailed first, and the modification of this algorithm will be proposed through the introduction of a new similarity measure (replacing the Euclidean distance, the widely used one with SOM applications) that is to be used to match different observations to their best matching neurons.

These algorithms will be presented by the help of a case study that recovers the missing and erroneous values in a set of remote sensing images (due to the presence of clouds and shadows). The application of the SOM algorithm to recover the missing data in a heavily cloudy region in France, monitor its superiority to other methods reported in the literature.

http://www.intechopen.com/download/pdf/10469

.