The goal of many algorithms in (biomedical) signal processing is, at the end of the day, to perform some sort of classification, e.g., binary classification. The binary labels (categorical response variables) could for instance represent the presence or absence of a disease or the binary signal quality of a physiological recording.
When working with time series data, such as the ECG, EEG, blood pressure etc., one can extract features from these signals, that are afterwards used for classification.
When working with the ECG for example, one could use the duration, height of the QRS complex etc., as features for a classification of normal/abnormal beats. Each training sample, corresponding to one heart beat, would consist of the feature vector and the label.
Now, when training a classifier, be it a SVM, Logistic Regression or some sort of decision tree/forest, the samples are clearly NOT independent. These classifiers however assume that the samples are IID and I observe that e.g. decision tree, overfit heavily on such data. This is also problematic with ensemble methods such as random forests which rely on bagging (resampling the training data).
What are common approaches to alleviate this problem or work with dependent samples in a classification procedure?
I know that alternative methods such as hidden Markov models are well suited for time series data, but I am specifically interested in the supervised classification setup using such type of learners (SVM, Log.Regression, ....).
Hi Federico,
here some ideas on your problem, which might be helpful: actually Hidden Markov Models can be applied in the sense of a supervised classifier as well and, in fact, they might be an appropiate solution for your problem.
Apart from that, there are other classifiers which take into account the temporal behaviour, you could for example have a look to recurrent neural networks.
If you want to use classifiers which do not take into account temporal dependencies it might be necessary to create features from the time courses you have. This "feature engineering" can be done manually, than using a priori knowledge (e.g. the biggest difference in QRS amplitudes from a certain time compared to the last 10 s could be one feature; such manually defined features might be better to process for a classifier than just passing all previous QRS amplitudes). Another way would be to extract features automatically, e.g. by convolutional neural networks, and use a supervised classifier afterwards.
Greetings, Sebastian
In addition to Sebastian's answer, I have seen probabilistic graphical models used to remove the IID assumption when using relational features which are measures of how samples are related. Some of these methods are extensions of Markov models or Gaussian networks. Here is a couple of resources that may help but a quick disclaimer, it may bring you down a bit of a exploration hole since this stuff gets dense:
http://pgm.stanford.edu/
http://psl.linqs.org/
Thank you very much to you all.
@ Sebastian: I am familiar with hidden Markov models and graphical models (we mostly use so called factor graphs, but Bayesian Networks or Markov random fields are not all too different from these). However, the way I have used HMM and saw them being used is in the context of dynamical systems (state space models) and estimation of parameters or the state itself (kalman filter/smoothing and variations of it or in the discrete case sth. like the Baum-Welch algorithm or Viterbi alg.).
Can you point me to some works (I guess you had some in mind when writing the answer), where HMM have been used explicitly for supervised binary classification?
Interesting comments regarding CNNs and RNNs. I am not very familiar yet with these techniques but will look into that.
However, I doubt I have enough data for learning both the features and learning the classifier. Unlike in, e.g., image processing problems such as the ImageNet competition, in my case, as oftentimes encountered in biomedical engineering, the sample sizes are rather small (~30 patients) since data collection in this field is more laborious than just taking a pic ;-)
Hi Federico,
the idea i referred to needs some training data, i.e. data were the classes and describing features are known. The classes are considered as hidden state, the features as emissions. Using this data you can "train" a HMM in a supervised way, i.e. determine the transition, emission and initial state probabilities. After you have created the model you can apply it to new data. In that data you only have the features and you can apply the viterbi algorithm in order to find the underlying hidden staes (classes).
Something similar was done in "Sleep Staging Based on Signals Acquired
Through Bed Sensor" by Kortelainen.
Greetings, Sebastian
- Badges
- Science topic
- Similar topics
- Biological Science
- Physiology
- Biosignals
- Biomedical Signal Processing